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FOREWORD 


This  research  and  development  was  conducted  under  contract  N00123-80-C-0847  with 
the  Human  Resources  Research  Organization.  This  effort  was  part  of  the  continuing 
project  in  support  of  exploratory  development  task  area  CF63-52 1-080- 102  (USMC 
Training  and  Education),  work  unit  03.06  (USMC  Professional  Military  Education).  The 
sponsor  was  the  Education  Center  of  the  Marine  Corps  Development  and  Education 
Command  (MCDEC),  Quantico,  Virginia.  The  project  was  initiated  as  the  result  of 
FY1980  proposed  exploratory  development  efforts  that  specified  a  need  for  (1)  new 
alternatives  to  traditional  group-paced  instructional  programs  for  resident  Marine  Corps 
professional  military  education  (PME)  and  (2)  new  strategies  in  nonresident  courses  that 
would  improve  student  motivation,  provide  interaction  with  peers  and  instructors,  and 
instructional  feedback. 

This  report,  the  second  in  a  series,  contains  the  instructional  package  developed  to 
assist  instructors,  who  are  graduates  of  an  individualized  instructor  training  program,  in 
the  individualization  of  their  own  courses.  The  first  report,  NPRDC  Special  Report  83-19, 
described  the  implementation  of  individualized  instruction  at  the  Instructional 
Management  School  that  trains  PME  course  instructors. 

The  contracting  officer's  technical  representative  was  Dr.  Michael  R.  Flaningam. 
The  recommendations  made  herein  are  directed  to  MCDEC  as  well  as  to  the  Chief  of 
Naval  Education  and  Training. 


3.  W.  RENARD 
Commanding  Officer 


3AMES  W.  TWEEDDALE 
Technical  Director 
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INTRODUCTION 


Problem 


Conventional,  lock-step,  group-paced,  lecture  courses  have  been  found  to  have  many 
shortcomings.  They  are  designed  to  permit  the  majority  of  a  class  to  accomplish  the 
objectives  of  the  course,  but  this  usually  means  that  the  course  is  designed  to  meet  the 
needs  of  the  hypothetical  average  student.  Group  testing  in  conventional  courses 
typically  does  not  provide  prescriptive  feedback  to  guide  the  individual  student  in 
subsequent  learning  activities. 

In  the  lock-step  mode,  there  is  little  allowance  for  initial  individual  differences  in 
prior  experience,  knowledge  level,  or  aptitude.  The  conventional  mode  does  not  allow 
individuals  to  proceed  at  their  own  optimum  rate  of  learning  and,  as  a  result,  the  fast 
learner  is  often  bored,  while  the  slow  learner  often  gives  up  in  despair. 

To  ameliorate  these  problems,  the  military  training  establishment  has  expended 
considerable  effort  to  individualize  much  of  its  instruction.  Much  of  this  effort  has  been 
performed  according  to  the  instructional  system  design  (ISD)  procedures.1  Unfortunately, 
much  of  the  available  ISD  documentation  is  quite  complex  and  not  sufficiently  specific  in 
some  areas  to  assist  instructors  in  the  practical  application  of  the  procedures  to  their  own 
courses. 

Background 

The  ISD  process  has  been,  and  continues  to  be,  applied  to  a  large  number  of  service 
schools.  However,  most  of  these  efforts  have  been  for  technical  training  or  skill-based 
activities.  Relatively  little  has  been  done  for  officer-level  professional  military  educa¬ 
tion  (PME)  courses,  which  are  heavily  cognitive  or  knowledge-based  in  nature. 

This  report  is  the  second  in  a  series  of  reports  covering  the  feasibility  of  individual¬ 
ization  for  PME  courses  in  the  Marine  Corps.  The  first  report2  reviewed  the  implemen¬ 
tation  of  an  individualized  course  at  a  instructional  management  school  (IMS)  for  training 
instructors  of  PME  courses.  The  intent  of  the  course  is  first  to  train  the  PME  instructors 
on  the  individualization  process,  and  then  to  assist  them  in  accomplishing  the  same 
process  for  their  own  courses. 

The  results  of  the  effort  at  the  IMS  were  very  positive.  Students  were  trained  for 
instructional  duties  more  efficiently  and  effectively  by  the  individualized  course.  As 
expected,  however,  additional  materials  were  still  needed  to  provide  guidance  to  the 
graduating  instructors  in  individualizing  their  own  courses.  Subsequent  effort  was  then 
directed  toward  developing  materials  that  would  help  these  instructors  to  use  ISD 
procedures  upon  their  return  to  their  own  schools. 


‘Chief  of  Naval  Education  and  Training.  Interservice  procedures  for  Instructional 
Systems  Development  (NAVEDTRA  106A).  Pensacola,  FLs  Author,  August  1975. 

2Flaningam,  M.  R.,  and  Joyner,  J.  N.  Feasibility  of  individualized  instruction  for 
USMC  professional  military  education  programs:  Phase  I.  Implementation  at  instruc¬ 
tional  management  school  (NPRDC  Spec.  Rep.  83-19).  San  Diego:  Navy  Personnel 
Research  ana  Development  Center,  March  1983.  (AD-A126  455) 
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Purpose 

The  objective  of  this  effort  was  to  develop  materials  to  assist  graduates  of  the  IMS  or 
similar  instructor  training  in  the  individualization  of  PME  courses. 

APPROACH 

The  materials  were  developed,  in  large  part,  on  the  basis  of  experience  and 
techniques  gained  from  implementing  individualized  instruction  at  the  IMS,  Quantico.  The 
emphasis  was  on  the  selection  and  development  of  materials  the'  would  enable  the 
instructors  to  exercise  individualization  principles  relevant  to  their  particular  courses. 
Existing  ISD  documents  were  also  reviewed  and  their  relevant  aspects  adapted  and 
appropriately  referenced. 

A  handbook  was  developed  to  guide  instructors  systematically  through  the  tasks  that 
should  be  accomplished  for  individualization.  An  associated  study  guide  provides  the 
instructors  with  opportunities  to  practice  needed  skills,  again  with  focus  on  the  their  own 
course.  Two  equivalent  tests  are  also  provided  for  the  instructors  to  take  and  use  as 
feedback  to  assist  them  in  identifying  areas  that  they  have  not  been  fully  understood. 

The  materials  are  designed  to  stand  alone,  based  upon  preparation  provided  at  an  IMS 
or  similar  instructor  training  facility.  However,  references  are  provided  for  users  desiring 
more  in-depth  coverage  of  certain  areas. 

RESULTS 

The  Handbook  for  Individualized  Instruction  and  its  associated  study  guide,  test 
forms,  and  course  forms  are  provided  in  the  appendix. 


RECOMMENDATIONS 


It  is  suggested  that  the  Education  Center  at  the  Marine  Corps  Development  and 
Education  Command  (MCDEC): 

1.  Implement  and  evaluate  the  effectiveness  of  these  documents  for  the  individual¬ 
ization  of  relevant  resident  PME  courses  at  MCDEC. 


2.  Consider  the  application  of  these  documents  to  individualize  the  nonresident 
equivalents  of  the  resident  PME  courses. 

It  is  suggested  that  the  Chief  of  Naval  Education  and  Training  (CNET): 

1.  Review  the  materials  for  potential  application  to  other  PME  courses  and 
evaluate  their  effectiveness  in  other  settings. 

2.  Review  and  evaluate  the  materials  for  application  to  technical  training  courses. 
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PREFACE 


Deciding  whether  or  not  to  use  individualized  instruction  is  part  of  the  instructional 
development  process.  This  decision  is  covered  in  Marine  Corps  Order  1553.1,  Phase  III.  In 
the  Interservice  Procedures  for  Instructional  Systems  Development  (NAVEDTRA  106A), 
this  decision  is  located  in  Block  III.  T:  "Specify  Instructional  Management  Plan  and 
Delivery  System."  This  text  defines  individualized  instruction  and  how  to  design  and 
manage  an  individualized  system. 

Before  you  study  this  text,  you  should  have  completed  an  instructor  course  or 
instructional  manager  course.  You  should  be  thoroughly  familiar  with  instructional 
systems  development  (ISD)  and  with  mastery  learning/teaching.  You  should  be  able  to 
perform  the  following  tasks: 

1.  Develop  objectives. 

2.  Develop  objective-referenced  tests. 

3.  Sequence  objectives. 

4.  Develop  a  lesson  outline. 

5.  Develop  a  self-paced  lesson. 

6.  Validate  instructional  materials. 

A  good  individualized  instruction  system  is  made  up  of  the  same  components  as  a 
good  lock-step  system:  presentations,  student  response  (frequent  practice),  feedback,  and 
evaluation.  It  should  be  based  on  sound  instructional  objectives.  The  main  difference 
between  individualized  and  nonindividualized  instruction  is  the  way  these  components  are 
put  together.  That  is  what  you  will  be  learning  about  in  the  following  chapters. 

This  handbook  is  aimed  at  those  who  want  to  use  individualized  instruction  in  their 
own  courses.  However,  much  of  what  the  handbook  contains  applies  to  all  instruction, 
including  traditional  lock-step  courses.  This  is  especially  true  about  evaluation.  Most  of 
the  rules  of  good  evaluation  apply  just  as  much  to  lock-step  courses  as  to  individualized 
courses.  Therefore,  you  may  find  useful  information  here  even  if  you  are  not  planning  to 
individualize. 

Appendix  A  is  a  study  guide  for  you  to  use  as  you  read  the  handbook.  Using  the  guide 
will  help  you  understand  and  remember  what  individualization  is  all  about. 

Appendix  B  provides  two  equivalent  tests  for  your  use  in  assessing  what  you  have 
learned  from  the  handbook. 

Appendix  C  contains  course  forms  for  you  to  use  with  or  without  revisions  in  your 
course. 
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INTRODUCTION 


"Individualized  instruction"  means  different  things  to  different  people.  What 
does  it  mean  to  you?  Programmed  texts?  Tape/slide  lessons?  Videotape?  More  attention 
given  to  students? 

Actually,  individualized  instruction  comes  in  many  forms.  Some  forms  use 
programmed  texts  but  most  do  not.  Some  use  audiovisual  media  such  as  tape/slide, 
videotape,  or  even  computers.  But  these  media  in  themselves  are  not  individualized 
instruction.  You  can  have  individualized  instruction  with  or  without  them. 

Individual  attention  to  students  is  a  little  closer.  Individualized  instruction 
always  pays  attention  to  individual  students.  But  it  is  more  than  just  "extra  help  if 
needed."  It  is  systematic  attention.  Individualized  treatment  of  students  is  built  right 
into  the  system  of  instruction. 

1.1  Instruction  Systems 

What  do  we  mean  by  system  of  instruction?  Just  this:  The  procedures  used  from 
the  time  the  student  begins  instruction  until  he  or  she  graduates.  That  is  a  system. 
Figure  1-1  provides  a  diagram  of  a  traditional  (not  individualized)  instruction  system  for 
teaching  American  history  in  college. 


Figure  1-1.  Example  of  nonindividualized  instruction  (System  A). 
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This  system  is  not  individualized  because  it  treats  all  students  the  same.  (It  is 
fair,  but  it  is  not  individualized.)  Figure  1-2  provides  a  diagram  of  a  more  individualized 
system.  In  the  individualized  system,  you  can  see  how  the  student  earned  credit  week-by¬ 
week.  The  standard  for  the  course  might  be  "student  must  master  13  subunits,*  14 
subunits  =  C,  15  subunits  =  B,  and  16  subunits  =  A." 


Figure  1-2.  Example  of  individualized  instruction  (System  B). 

System  B  is  more  individualized  than  system  A  because  different  students  get 
different  treatments.  The  students  who  do  not  pass  the  test  after  the  lecture  receive 
extra  assignments  and  attend  extra  classes.  They  are  not  treated  the  same  way  as  other 
students. 


1.2  Performance  Counts 


In  system  B,  students  are  treated  individually  according  to  performance — usu¬ 
ally,  according  to  how  they  perform  on  tests  or  quizzes.  In  system  B,  for  example, 
students  who  pass  die  first  test  get  credit  for  one  subunit  and  do  not  have  to  go  to  any 
more  classes  or  do  any  more  assignments.  Differences  among  the  students  will  show  up  in 
their  performance  in  the  course.  Here  are  some  of  the  ways  that  students  differ: 

1.  Previous  knowledge. 

2.  Aptitude  for  learning. 

3.  Motivation  to  learn. 

4.  Study  habits. 

5.  Writing  skill. 

6.  Reading  skill. 

There  are  others,  of  course,  but  these  are  enough  to  think  about  for  now. 
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How  are  you  going  to  treat  good  readers  differently  from  poorer  readers?  How 
are  you  going  to  treat  students  who  already  know  a  lot  about  your  subject?  What  about 
students  with  low  motivation? 

The  answer  to  all  these  questions  is  the  same:  Use  individualized  instruction. 
However,  individualized  instruction  comes  in  many  forms.  There  are  many  individualized 
systems  ail  ready  in  use.  Some  of  these  will  be  described  in  Chapter  2.  Right  now,  we  will 
look  at  some  of  the  things  you  can  vary  to  individualize  instruction.  In  an  individualized 
system,  at  least  one  component  of  instruction  usually  varies  from  student  to  student. 

1.3  Variable  Time 

Nearly  all  individualized  systems  use  variable  time.  By  allowing  students  to 
study  for  different  periods  of  time,  you  automatically  allow  for  many  of  the  ways  that 
students  differ.  Students  who  know  the  subject  go  faster.  Students  with  a  lot  more  to 
learn  take  more  time.  Students  with  high  aptitude,  high  motivation,  and  good  study  habits 
and  skills  learn  faster.  Students  with  low  aptitude,  low  motivation,  and  poor  habits  and 
skills  will  be  slower. 

These  learning  differences  are  very  obvious,  but  many  instruction  systems  do  not 
do  anything  about  it.  In  nonindividualized  instruction  systems,  the  slower  students  do  not 
get  more  time.  They  just  learn  less.  An  individualized  system  usually  allows  time  to  vary 
from  student  to  student. 

This  does  cause  some  problems.  For  example,  does  a  student  who  is  not 
motivated  have  an  unlimited  amount  of  time?  No.  Notice  how  this  problem  was  solved  in 
system  B.  Although  slower  students  were  given  more  time  to  go  to  more  classes,  there 
was  a  limit  of  1  week.  Any  student  who  did  not  pass  a  weekly  test  by  the  end  of  the  week 
did  not  get  credit  for  the  subunit. 

That  is  just  one  example.  There  are  many  other  ways  to  combine  variable  time 
with  time  limits.  The  decision  of  how  much  time  to  allow  is  part  of  designing  your 
individualized  system.  You  probably  will  have  to  go  through  a  period  of  trial  and  error 
before  you  get  it  right. 

Another  problem  caused  by  variable  time  is  what  the  faster  student  Should  do 
while  the  slower  students  are  still  working.  In  system  B,  a  student  who  passed  the  weekly 
test  right  after  the  lecture  on  Monday  was  not  required  to  attend  class  on  Wednesday  or 
Friday.  That  is  fine  for  an  American  history  class  in  college  because  students  who  are 
good  in  American  history  can  use  the  extra  time  to  study  other  subjects  that  are  harder 
for  them  or  study  history  at  a  more  advanced  level. 

However,  system  B  would  not  always  work  In  military  training.  If  you  are 
training  persons  for  their  first  job,  you  do  not  want  to  waste  time  by  giving  them  a  lot  of 
time  off.  It  would  be  better  to  set  up  a  self-paced  system  in  which  students  move  on  as 
soon  as  they  pass  the  previous  lesson.  This  allows  the  faster  students  to  graduate  early 
and  get  on  the  job. 

In  other  cases,  it  might  be  acceptable  to  allow  a  student  "free  time."  At  an 
advanced  school  like  the  Command  and  Staff  College,  faster  students  might  be  able  to  use 
extra  time  for  independent  study.  There  is  no  set  formula.  You  will  be  looking  at  various 
possible  individualized  systems  in  the  next  chapter.  The  best  system  for  you  will  depend 
on  the  type  of  course  you  teach  and  what  your  students  could  do  with  extra  time. 
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1.4  Variable  Method 

System  B  (Figure  1-2)  also  illustrates  one  way  to  vary  the  method  of  instruction 
by  using  the  following  four  different  instructional  methods  or  treatments: 

1.  Reading  assignments. 

2.  Lectures. 

3.  Small-group  classes. 

4.  Extra  assignments. 

Not  all  students  receive  all  methods.  Students  who  passed  the  test  on  Monday  received 
only  treatments  1  and  2.  Students  who  failed  Monday's  test  received  treatments  1  and  2 
followed  by  treatment  3.  Students  who  failed  Wednesday's  test  received  all  four 
treatments. 

This  is  only  one  way  to  vary  the  instructional  method.  There  are  many  other 
ways.  You  may  think  of  some  yourself.  Again,  there  is  no  single  formula  to  follow.  The 
way  you  vary  instructional  methods  will  depend  on  what  you  are  teaching  and  what  your 
students  are  like. 

The  purpose  for  using  different  instructional  methods  is  always  the  same — to 
give  more  help  to  students  who  need  it.  You  do  not  want  to  spend  time  giving  one-to-one 
assistance  to  a  student  who  does  not  need  help.  Maybe  a  written  presentation  will  do. 
Other  students  may  need  more  than  a  text.  Controlling  the  methods  of  instruction  is  one 
of  the  ways  to  build  an  individualized  system. 

1.5  Media 

There  are  many  different  media  for  presenting  information:  the  printed  word,  a 
photograph,  a  drawing,  a  live  voice,  a  recorded  voice,  a  film,  a  videotape,  and  more.  The 
medium  of  communication  is  another  factor  you  can  vary  in  individualizing  your 
instruction.  For  example,  if  you  want  to  ask  your  students  a  question,  you  could  ask  it 
aloud  during  a  lecture.  Or  you  could  ask  it  in  a  printed  text,  workbook,  or  videotape. 

Suppose  you  want  to  demonstrate  good  and  bad  listening  habits.  You  could 
develop  a  videotape  lesson,  a  live  lecture/demonstration,  a  tape/ slide  lesson,  or  what  have 
you.  Can  you  think  of  other  media? 

The  media  most  often  used  for  instruction  are  the  printed  word  and  live 
presentations  such  as  lectures,  seminars,  conferences,  and  demonstrations.  However,  as 
more  courses  are  being  individualized,  instructors  are  turning  away  from  live  presen¬ 
tations  and  relying  more  on  print  and  other  media,  because  live  presentations  have  two 
important  drawbacks: 

1.  They  are  not  easily  repeated. 

2.  They  are  not  easily  referred  to. 

Once  a  live  lecture  has  been  given  to  a  group  of  students,  it  is  over.  It  is  not 
practical  to  give  it  again  to  more  students  the  next  hour  or  day.  No  instructor  would  want 
to  deliver  the  same  lecture  over  and  over  again,  nor  would  it  be  a  good  use  of  the 
instructor's  time.  That  is  one  of  the  differences  between  the  two  instruction  systems  we 
are  discussing.  System  B  used  only  one  lecture,  instead  of  three.  For  students  who  had 
not  mastered  the  lesson  after  Monday's  lecture,  system  B  used  other  methods:  extra 
exercises  and  small-group  classes.  It  would  not  be  practical  to  give  the  lecture  over  and 
over. 
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System  B  is  just  an  example.  A  lot  of  other  individualized  instruction  systems  do 
not  use  live  presentations  at  all.  They  use  the  instructor  as  a  personal,  one-to-one  tutor, 
like  a  coach.  To  present  the  information  initially,  these  systems  use  the  printed  word  or 
audiovisual  media  like  tape/slide.  A  printed  text — like  the  one  you  are  reading — can  be 
given  to  students  when  they  need  it,  as  can  a  workbook — like  the  ones  used  at  the  Marine 
Corps  Instructional  Management  School  (IMS)  to  teach  how  to  develop  instruction. 

Print  and  audiovisual  presentations  can  also  be  referenced,  and  students  can 
reread  and  restudy  them  as  often  as  necessary.  This  is  not  true  with  live  presentations. 

(Students  can  look  at  their  lecture  notes,  but  that  is  not  the  same.) 

The  kind  of  media  you  should  use  in  your  individualized  system  depends  on  the 
kind  of  system  you  design.  You  may  design  a  system  that  uses  live  presentations  or  uses 
instructors  only  as  one-to-one  tutors.  You  may  also  develop  tape/ slide  lessons  or 
videotape  lessons.  We  will  discuss  different  systems  in  the  following  chapters. 

Note.  The  ideal  system.  In  theory,  you  need  to  wait  to  see  how  students 
perform  in  your  course.  You  could  test  them  before  instruction  begins  to  learn 
their  aptitudes,  interests,  and  special  skills- -"learning  style."  Then,  you  could 
match  different  instructional  methods  to  different  students. 

In  theory,  students  who  learn  best  from  books  would  read  books,  students 
who  learn  best  from  lectures  would  get  lectures,  fast  students  would  get  less 
time  to  study,  slow  students  would  get  more  time,  and  so  on.  All  students  would 
be  treated  according  to  how  they  learn  best.  This  is  the  "ideal"  individualized 
system.  From  the  point  of  view  of  getting  everybody  through  the  system  in  the 
most  efficient  way,  it  is  the  best.  Unfortunately,  such  a  system  is  still 
theoretical.  We  do  not  know  enough  about  the  way  people  learn  to  match 
methods  to  students.  Maybe  we  never  will.  * 

What  "type"  of  student  will  learn  best  from  tape/slide?  We  are  not 
certain.  Questions  like  this  are  still  research  questions.  Until  scientists  can  tell 
us  more  about  what  makes  us  tick,  we  will  have  to  be  satisfied  with  the  best 
practical  system  we  can  build.  In  actual  practice,  the  best  we  can  do  is  build  an 
instruction  system  that  treats  students  according  to  performance. 

1.6  Summary 

The  purpose  of  individualizing  instruction  is  to  help  students  learn.  Because 
their  learning  ability,  knowledge,  skill,  and  motivation  differ,  they  do  not  leam  the  same 
amount  at  the  same  rate.  It  is  safe  to  say  that,  if  your  instruction  treats  all  your  students 
alike,  it  will  not  be  as  effective  as  if  you  set  up  a  system  that  treats  students  individually 
by  giving  them  different  amounts  of  time  to  learn,  various  instruction  activities,  or 
various  media. 

This  handbook  tells  you  how  to  design  and  set  up  such  a  system.  Chapter  2 
discusses  several  individualized  systems;  Chapter  3,  the  instructor's  role  in  Individualized 
instruction;  Chapter  4,  designing  an  individualized  Instruction  system  to  fit  your  needs; 

Chapter  5,  the  management  and  control  of  individualized  instruction;  Chapter  6,  the 
importance  of  evaluation  in  individualization;  Chapters  7  and  8  respectively,  how  to  plan 
and  develop  tests;  and  Chapter  9,  how  to  try  out  and  revise  tests. 

The  quality  control  procedures  In  Section  5.1  are  based  on  the  Instructional  - 

Quality  Inventory:  II.  User's  Manual  (EUls,  Wulfeck,  8c  Fredericks,  1979).  The  procedures 
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for  developing  performance  tests  (Section  8.3)  and  for  test  tryouts  (Sections  9.1  and  9.2) 
are  based  on  workshop  materials  for  the  development  and  validation  of  skill  qualification 
tests  (Campbell,  Ford,  &  Campbell,  1978). 

Appendix  A  provides  a  study  guide  for  your  use  as  you  study  this  handbook. 
Appendix  B  contains  two  forms  of  a  test  for  you  to  take  to  assess  what  you  have  learned 
in  this  handbook.  Appendix  C  contains  many  of  the  course  forms  discussed  throughout  this 
handbook.  You  may  reproduce  these  forms  as  is  or  revise  as  needed  for  your  course. 

2.0  APPROACHES  TO  INDIVIDUALIZED  INSTRUCTION 

All  approaches  to  individualizing  instruction  are  not  the  same.  There  are  many 
choices  and  the  choice  is  yours.  However,  you  will  probably  want  to  model  your 
individualized  system  after  one  that  you  already  know  about.  One  model  is  the  system 
used  for  most  of  the  lessons  at  IMS,  which  will  be  described  in  this  chapter. 

Another  system  is  the  instructional  package  you  have  in  your  hands,  which  does 
not  use  workbooks.  It  uses  a  study  guide  instead.  You  may  want  to  model  your  system 
after  it. 


Several  other  models  of  individualized  instruction  will  also  be  discussed  in  this 
chapter.  As  you  read  about  these  systems,  think  how  each  system  would  work  for  your 
course.  Would  it  work  with  your  students?  Your  subject?  Your  classroom  facilities? 
Better  yet,  think  about  how  you  might  change  some  of  the  things  in  your  course.  Maybe 
rearranging  your  classroom  would  allow  mofe  individualization.  In  other  words,  do  not 
take  everything  in  your  present  course  as  a  "given."  Things  can  be  changed— including  the 
way  people  think  about  individualization.  It  has  happened  before. 

Before  you  continue,  here  is  a  reminder.  If  you  still  think  that  individualized 
instruction  only  means  "programmed  texts"  or  "tape/slide"— or  any  other  single  method— 
forget  it  now.  There  is  a  lot  more  to  individualizing  than  that.  To  find  out  what,  keep 
reading. 


2.1  Four  Questions 

A  good  way  to  think  about  individualized  systems  is  in  terms  of  the  four 
questions  below.  As  you  look  at  several  different  systems  in  the  rest  of  this  chapter,  keep 
these  questions  in  mind. 

1.  Do  all  students  study  the  same  CONTENT?  In  many  individualized  systems, 
all  students  study  the  same  subject  matter  with  the  same  objectives.  In  other 
individualized  systems,  the  more  advanced  students  study  more  advanced  content  or  study 
"electives."  Which  would  be  better  for  your  course? 

2.  Are  all  students  allowed  the  same  amount  of  TIME?  In  Chapter  1,  we 
pointed  out  that  most  individualized  systems  allow  time  to  vary  student-by-student. 
Students  who  have  more  knowledge,  motivation,  or  learning  aptitude  move  through  the 
course  faster  and  graduate  sooner  than  do  others.  In  other  individualized  systems,  all 
students  are  allowed  the  same  amount  of  time;  for  example,  an  independent  study 
program  in  which  students  might  be  allowed  a  fixed  amount  of  time  to  study  in  their  own 
area. 
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3.  Are  all  students  given  the  same  instruction  METHOD?  In  Chapter  1 ,  we  saw 
the  combination  of  methods  used  in  system  B  to  individualize  Instruction.  Students  who 
did  not  learn  well  from  one  method  were  given  another.  In  system  B,  the  methods  were 
lecture,  extra  assignments,  and  small-group  classes.  Another  system  might  combine 
tape/slide  with  one-to-one  tutorial,  programmed  texts  with  lectures,  or  you  name  it. 
What  methods  would  work  best  in  your  course? 

4.  Do  all  students  reach  the  same  level  of  ACHIEVEMENT?  You  may  call  this 
"level  of  proficiency"  or  "level  of  mastery."  It  all  means  the  same  thing.  In  most 
individualized  systems,  there  is  a  single  level  of  achievement  ("mastery")  for  all  students. 

In  other  systems,  students  graduate  with  different  levels  of  proficiency.  For 
example,  in  an  independent  study  program  in  which  study  time  is  fixed,  faster  students 
might  learn  more,  although  all  students  might  be  required  to  meet  a  minimum  standard. 
In  general,  if  content,  time,  and  method  do  not  vary,  level  of  achievement  will.  Some 
students  will  just  learn  more  than  others.  It  is  human  nature. 

Now  let  us  look  at  some  real-life  individualized  systems.  We  will  classify  each 
system  according  to  content,  time,  method,  and  achievement.  As  you  read  about  these 
systems,  keep  thinking  about  what  kind  of  system  would  be  best  for  your  situation. 

2.2  Workbook/Tutorial  System 

The  individualized  system  in  use  at  IMS  has: 

1.  Variable  content. 

2.  Variable  time. 

3.  Fixed  method. 

4.  Fixed  achievement  level  ("mastery"). 

We  say  that  content  is  variable  because  the  lesson  students  study  depends  on 
their  duty  assignments.  Although  most  IMS  students  study  the  lessons  in  the  instructor 
track,  others  may  study  only  a  few  of  these  and  concentrate  on  lessons  about  task 
analysis.  Others  may  concentrate  on  evaluation.  It  all  depends  on  what  they  need  to 
learn  for  their  job. 

The  IMS  system  is  a  variable  time  system  because  performance  on  tests,  not  a 
schedule  or  the  calendar,  determines  student  progress.  Some  students  move  through  their 
lessons  more  rapidly  them  do  others,  because  they  already  know  some  of  the  subject 
matter,  they  are  "fast  learners,"  or  maybe  they  are  in  a  hurry.  The  students  who  enter 
IMS  are  different  in  many  ways;  for  example,  instructional  experience,  years  in  service, 
officer/enlisted,  military  occupational  specialty  (MOS),  years  of  education,  etc.  So  it  is 
not  surprising  that  they  do  not  progress  at  the  same  rate. 

The  IMS  system  is  a  fixed  method  system  because  all  students  are  given  the  same 
treatment:  lesson  workbooks  with  one-to-one  instructor  assistance.  Of  course,  the 
amount  of  assistance  varies  from  student  to  student.  Some  students  need  a  lot  of  help. 
Others  "whiz"  through.  The  system  is  still  called  "fixed-method"  because  students  who 
fail  to  master  a  lesson  objective  continue  to  learn  using  the  same  methods:  workbooks, 
tutorial  help  from  the  instructor,  practical  exercises,  and  rehearsed  presentations. 
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vidualized  systems: 


1.  Short  lessons  (modules). 

2.  Frequent  tests. 

3.  Testing  to  "mastery”  level 
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4.  Self-pacing  (variable  time). 

5.  Printed  workbooks  instead  of  lectures. 

2.3  Study  Guide  System 

You  are  using  an  individualized  system  right  now.  The  text  you  are  holding  is 
part  of  an  instructional  package.  The  other  parts  of  the  package  are  a  study  guide  and 
tests.  Although  the  instructional  package  was  developed  to  be  used  at  IMS,  it  is  self- 
contained  and  could  be  used  anywhere. 

How  does  the  study  guide  system  differ  from  the  workbook/tutorial  system  used 
for  most  of  the  lessons  at  IMS?  One  difference  is  that  the  study  questions  are  printed 
separately  in  the  study  guide,  instead  of  right  along  with  the  lesson  ir  a  workbook. 

Another  difference  is  that  there  is  more  to  read  ki  the  study  guide  system.  In 
the  workbook/tutorial  system,  short  sections  of  text  are  followed  immediately  by 
questions,  which  is  where  the  "tutorial"  part  comes  in.  The  instructor  and  the  workbook 
together  instruct  the  student.  The  instructor  does  not  put  a  whole  lot  of  information  into 
the  workbook.  The  workbook  just  gives  enough  to  get  the  student  started.  The  rest  is  up 
to  the  instructor.  In  other  words,  the  workbook/tutorial  system  could  never  work  if  an 
instructor  were  not  on  hand  at  all  times.  It  is  not  as  self-contained  as  the  study  guide 
system. 


The  study  guide  system  prints  more  information.  The  student  has  more  reading 
to  do.  Although  students  who  do  not  understand  something  in  the  text  can  always  ask  for 
help,  in  general,  the  text  contains  the  information  students  need. 

One  advantage  of  the  study  guide  system  is  that  you  can  use  existing  textbooks 
or  other  publications  like  FMFMs.  First,  you  would  find  one  or  more  publications 
containing  the  subject  matter  your  students  need  to  learn.  Then,  you  would  write  a  study 
guide  with  lesson  objectives,  guidance  on  different  parts,  and  study  questions.  You  would 
develop  only  the  study  guide,  not  the  "source  publications." 

Are  there  any  source  publications  for  your  course?  Is  the  information  that  your 
students  need  to  learn  already  printed  in  a  manual,  order,  textbook,  or  other  publication? 
If  so,  then  you  could  use  the  study  guide  system  to  individualize  your  course. 

The  study  guide  system  has  these  features: 

1.  Fixed  content  (text). 

2.  Variable  time. 

3.  Fixed  method  (study  guide). 

Achievement  level — either  fixed  or  variable. 

Figure  2-2  shows  how  a  student  would  progress  through  a  study  guide  system  if 
achievement  level  were  fixed. 

The  study  guide  system  and  the  workbook/tutorial  system  are  a  lot  alike;  both 
use  printed  books  to  teach  the  subject.  Since  the  study  guide  system  requires  more 
reading,  it  might  be  better  for  students  who  are  more  advanced  or  better  able  to  study 
independently.  It  is  also  a  good  bet  if  you  already  have  a  good  source  publication  that 
contains  what  the  students  need  to  learn. 
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Note.  A  good  source  publi 
just  to  save  the  trouble  of  \ 
textbook  is  poor,  your  cours< 
know  it. 


Figure  2- 


2.4  Fixed-pace  Individualization 

Fixed-pace  and  individualize 
alized  systems  are  entirely  self-paced 
a  fixed  schedule  of  lectures,  audiovi 
allowing  both  achievement  level  and  ir 
these  features: 

1.  Fixed  content. 

2.  Variable  time  (to  study). 

3.  Fixed  method. 

4.  Variable  achievement. 


To  individualize  within  a  fix< 

1.  Present  instruction  on  a 

2.  Give  your  students  time 


3.  Give  all  students  a  test  at  a  scheduled  time. 

4.  Require  students  who  fail  the  test  to  restudy  the  material. 

5.  Retest  the  students  who  failed  the  first  test. 

This  way,  students  who  need  more  time  to  study  can  take  it  and  students  who  have  not 
mastered  the  subject  can  be  retested  until  they  do.  Extra  study  and  retest  time  are 
probably  not  possible  in  MOS-producing  courses.  However,  it  probably  is  available,  if  you 
are  teaching  a  course  in  the  area  of  professional  military  education.  Figure  2-3  is  a 
diagram  of  a  fixed-pace  program. 


6.  Student*  ate  rateeted.  Thoae  who 
fell  continua  in  tha  cycla. 


Figure  2-3.  Fixed-pace  individualization. 


In  a  fixed-pace  program,  when  there  are  no  more  presentations,  the  course  is 
over.  Different  students  will  have  spent  more  or  less  time  studying  before  the 
presentations,  and  especially  afterwards.  These  who  prepared  well  for  the  presentations 
and  paid  more  attention  during  the  presentation  are  more  likely  to  pass  the  test  the  first 
time.  Students  who  did  not  read  their  advance  sheets  are  less  likely  to  pass.  They  have  to 
make  up  for  this  by  studying  more  after  the  presentation. 

In  this  system,  the  .evels  of  achievement  may  vary.  If  some  students  have  not 
mastered  all  the  lessons  by  the  time  the  course  ends,  the  course  mastery  criterion  might 
be  a  proportion  of  lessons  mastered;  for  example,  14  out  of  16.  You  could  even  make  it 
100  percent,  as  long  as  you  are  prepared  f^r  some  students  not  meeting  that  criterion  in 
the  fixed  time. 
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Remember,  you  can  individualize  a  course  with  a  fixed  schedule  of  lectures, 
audiovisual  presentations,  guest  lectures,  or  whatever.  The  real  meaning  of  individu¬ 
alization  is  not  just  "self-pacing."  Individualization  means  treating  individual  students 
differently,  which  you  can  do  even  when  you  teach  lessons  on  a  fixed  schedule. 

2.5  Programmed  Instruction 

You  could  build  a  system  around  programmed  workbooks.  Programmed  instruc¬ 
tion  is  based  on  breaking  the  subject  down  into  very  small  parts  (see  Figure  2-4).  The  idea 
is  to  "feed"  students  just  a  little  bit  of  content  at  a  time,  to  check  to  see  if  they 
understand  it,  and  immediately  tell  them  if  they  are  right  or  wrong. 


The  amphibious  assault  establishes 
a  force  on  a  hostile  shore.  The 
amphibious  withdrawal  removes  forces 
from  a  hostile  shore. 

The  amphibious  operation  that  estab¬ 
lishes  a  force  on  a  hostile  shore 
is  the  amphibious 

assault 

Figure  2-4.  Example  of  programmed  instruction. 


Programmed  instruction  tries  to  keep  students  making  correct  responses  because 
it  goes  only  a  small  step  at  a  time.  The  programmed  instruction  reinforces  the  right 
answer  by  giving  students  the  answer  right  away. 

Just  like  other  kinds  of  instruction,  a  well-written  programmed  workbook  can  be 
very  effective.  Remember  that  programmed  instruction  is  not  an  individualized  system. 
You  still  have  to  build  a  system  to  use  programmed  texts. 

Figure  2-5  shows  how  programmed  texts  can  be  used  in  an  individualized  system 
of  instruction.  The  individualized  system  just  described  can  be  characterized  by: 

1 .  Fixed  content. 

2.  Variable  time. 

3.  Variable  method  (programmed  text  +  ?). 

4.  Fixed  achievement. 

2.6  Worksheet  System 

An  individualized  system  that  has  been  used  successfully  in  Army  training  may 
be  called  the  worksheet  system.  In  this  system,  there  are  no  textbooks,  workbooks, 
lectures,  videotapes,  or  any  other  presentation  media.  The  entire  system  is  built  around 
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Lesson  in 

programed 

text  _ 


Student  restudles 
selected  parts 
of  text 


1.  The  student  reads  a  lesson  in 
the  programmed  text. 

2.  Test. 

3.  If  the  student  passes  the  test, 
he  or  she  studies  another 
lesson. 

4.  If  not,  the  student  could 
restudy  the  parts  of  the 
programmed  book  that  cover 
the  errors  on  the  test. 

5.  Retest 


6.  If  the  student  failed  the  retest, 
it  is  doubtful  that  studying  the 
programed  text  any  more  would 
help,  since  he  or  she  has  already 
answered  the  questions  in  it  twice 
Some  other  method  should  be  used 
at  this  point. 


Figure  2-5.  Example  of  individualized  system  of  instruction  using 
programmed  texts. 


practical  exercises.  Students  take  a  pretest  and  then  (unless  they  pass)  receive  worksheet 
set  if  1  with  lots  of  practice.  Before  students  are  allowed  to  retake  the  test,  they  must 
produce  the  correct  answer  to  all  the  questions  on  all  the  worksheets  in  the  set.  Any 
students  who  fail  the  test  a  second  time  get  worksheet  1t2  and  the  cycle  continues.  For 
students  who  fail  a  third  test,  the  instructor  must  decide  whether  they  should  go  on  to  the 
next  lesson  or  get  more  practice. 

As  no  information  is  presented  in  this  system,  it  can  only  be  used  to  practice 
something  that  the  students  already  know  (basically)  how  to  do.  For  example,  processing 
supply  requests  is  mostly  a  matter  of  following  specific  directions  in  a  publication.  It  just 
takes  practice  and  being  careful.  The  worksheet  system  of  individualized  instruction  is  a 
good  way  to  build  up  proficiency  at  such  tasks.  Figure  2-6  shows  how  a  student  moves 
through  a  lesson  in  the  worksheet  system. 

The  worksheet  system  can  be  classified  this  ways 

1.  Fixed  content. 

2.  Variable  time. 

3.  Fixed  method  (practical  exercise). 

4.  Fixed  achievement. 
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@  Practice 
Worksheet  Set  #1 


©  Practice 

Workaheet  Set  #2 


142.  The  atudent  take*  a  preteat. 

If  he  or  ahe  paaaea  (901) ,  the 
atodenta  aklpa  the  lesson. 

3.  In  the  Any  prograa,  practice 
workaheet  act  #1  conalata  of  ten 
vorkaheeta  with  ten  items  each, 
for  a  total  of  100  Items.  The 
atudent  Mat  neater  all  ltsms 
before  taking  the  teat  again.  A 
atudent  grader,  or  proctor,  grades 
the  worksheet. 

4.  Test. 

5.  Any  atudent  who  falls  the  test, 
receives  practice  worksheet  set 
f2,  which  contains  100  new  Items. 

6.  Test. 

7.  If  the  student  fall  the  test  a 
third  tine,  the  lnstvuctor  mat 
decide  whether  to  send  the  stu¬ 
dent  to  the  next  lesson  (without 
mastery)  or  assign  more  practice. 


Figure  2-6.  Worksheet  system. 

Only  time  varies.  Although  the  instructor  is  always  available  to  help  students  during  the 
practical  exercises,  each  student  has  to  master  each  exercise. 

2.7  Peer  Training  System 

Peer  training  or  the  "buddy  system"  is  an  individualized  system  used  in  self- 
paced  MOS-producing  courses.  This  system  features  a  series  of  hands-on  performance 
stations  that  every  trainee  must  pass.  At  each  station,  an  instructor  does  the  testing. 

This  particular  system  also  uses  other  methods  and  presentation  media.  Besides 


2 ■  The  atudant  practices  the  taak 
(«uch  aa  operating  a  VHF  radio) 
with  a  partner. 

3.  Hands-on  performance  teat. 

4.  Aa  aoon  aa  a  atudant  paaaaa,  ha  or  aha 
goaa  to  tha  next  laaaon.  If  ona 
atudant  paaaaa  tha  teat  and  hie  or  har 
partner  fails,  tha  remaining  atu¬ 
dant  finda  a  new  partner. 

3.  Tha  cycle  continues  with  more 
practice  until  the  atudant 

maatara  tha  teak. 


Figure  2-7.  Peer  training  system. 


1.  Workbook  tutorial 

2.  Study  guide 

3.  Fixed  pace 

4.  Programmed  text 

5.  Worksheet 

6.  Peer  training 


F  =  fixed 
V  =  variable 
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Figure  2-8.  Summary  of  six  individualization  systems 
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We  have  limited  our  examination  to  systems  you  can  build.  There  are  many 
other  individualized  systems  that  we  have  not  discussed,  such  as  the  computer-managed 
instruction  (CM1)  used  at  several  Navy  "A"  schools,  the  Marine  Corps  Institute  nonresident 
training  system,  and  the  personalized  system  of  instruction  (PSD  used  at  some  colleges 
and  by  the  Naval  Postgraduate  School  Office  of  Continuing  Education.  If  you  individu¬ 
alize  your  course,  there  will  be  one  more  system.  Before  you  continue,  pause  for  a 
moment  and  think  how  you  might  individualize  your  own  course.  Consider  each  of  the 
four  factors  below  and  say  to  yourself  whether  you  would  arrange  for  each  factor  to  be 
fixed  or  variable: 

1.  Content — fixed  or  variable? 

2.  Time— fixed  or  variable? 

3.  Method— fixed  or  variable? 

k.  Achievement  level— fixed  or  variable? 

2.9  Audiovisual  Media 

"Audiovisual"  is  a  term  used  to  describe  certain  media  of  instruction,  not  a 
system  of  instruction.  Any  of  the  systems  we  have  described  could  use  audiovisual 
materials.  Tape/ slide,  film  strip  (like  cue-see),  and  videotape  programs  are  the  most 
likely  choices  because  they  are  all  relatively  easy  to  produce.  (Motion  picture  production 
is  more  difficult.)  Tape/slide  is  the  easiest  audiovisual  material  to  update  or  revise  and, 
if  you  do  not  have  access  to  videotape  or  photographic  production  equipment,  the  least 
expensive. 

If  you  have  videotape  equipment,  you  can  use  it  over  and  over  without  much 
additional  cost  except  for  maintenance  and  repair.  If  you  have  access  to  a  photography 
laboratory,  filmstrips  are  fairly  inexpensive  to  make.  As  far  as  cost  goes,  tape/slide, 
videotape,  and  filmstrip  programs  are  all  cheaper  to  produce  than  are  motion  pictures. 
Almost  anything  in  a  workbook,  text,  or  lecture  can  be  put  into  an  audiovisual  format  with 
planning  and  time.  The  procedure  for  developing  an  audiovisual  script  is  explained  in  the 
instructional  management  course  lesson  on  developing  tape/slide  lessons. 

The  main  advantage  of  audiovisual  materials  is  just  what  the  name  sug¬ 
gests — sight  and  sound.  Your  students  hear  and  see  the  subject  at  the  same  time,  which 
may  help  them  to  remember.  In  general,  you  should  use  audiovisual  media  when  they  are 
more  efficient  than  other  means  such  as  texts  or  lectures.  A  basic  lesson  on  multichannel 
radio  systems  might  be  a  good  example.  If  you  needed  to  teach  officers  and  staff  NCOs 
at  many  levels  about  multichannel  equipment,  using  a  tape/slide  or  videotape  presentation 
might  be  more  efficient  than  to  repeat  the  same  illustrated  lecture  over  and  over.  To  get 
the  benefits  of  audiovidual  materials,  you  have  to  illustrate  your  subject.  Showing  a  lot  of 
words  on  a  screen  and  reading  them  to  the  students  is  not  much  better  than  giving  them  a 
book.  (It  may  not  be  as  good  because,  with  a  book,  students  can  quickly  reread  what  they 
have  read.) 

If  you  use  audiovisual  presentations,  remember  that  you  still  need  all  the  other 
parts  of  an  instruction  system.  Audiovisual  media  are  just  ways  to  communicate 
information.  If  you  are  going  to  use  videotape,  for  example,  you  still  need  to  design  a 
system  that  will  use  videotape. 


Figure  2-9  shows  how  am  audiovisual  presentation  can  be  added  to  a  system  that 
uses  a  programmed  text.  In  this  system,  students  who  fail  the  first  test  can  study  the 
audiovisual  lesson  to  get  ready  for  the  retest.  (Of  course,  we  could  have  given  the 
audiovisual  lesson  first  and  saved  the  programmed  text  for  later.)  Students  who  fail  the 
second  test  (the  retest)  could  restudy  the  programmed  text,  the  audiovisual  lesson,  or 
both. 


Figure  2-9.  Example  of  the  use  of  audiovisual  presentations  in  an 
individualized  system. 


2.10  Summary 

There  are  many,  many  ways  to  individualize  a  course.  You  can  vary  the  content, 
the  time,  the  instruction  method,  and  the  student's  achievement  level,  all  four,  or  any 
combination.  In  this  chapter,  you  saw  several  systems  illustrated.  By  now  you  should 
have  some  ideas  about  how  your  own  course  could  be  individualized.  Remember  that  you 
do  not  have  to  copy  any  system  "as  is."  Borrow  ideas  as  freely  as  you  can  to  create  an 
approach  that  fits  your  situation.  Finally,  remember  that  audiovisual  media  can  make 
your  course  more  effective.  You  have  to  build  an  individualized  system  for  the 
audiovisual  media  to  fit  into  because  audiovisual  media  themselves  are  not  substitutes  for 
an  individualized  instruction  system. 

In  this  chapter,  we  also  mentioned  the  role  of  the  instructor  from  time  to  time. 
The  instructor's  role  in  an  individualized  system  is  generally  more  demanding.  The  next 
chapter  tells  you  why. 
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3.0 


THE  INSTRUCTOR’S  ROLE 


A  mistaken  idea  some  people  have  about  individualized  instruction  is  that  you 
can  "get  rid  of  the  instructor"  when  you  individualize  a  course.  The  truth  is  that  the 
instructor  is  more  important  than  ever  and  will  also  have  a  lot  more  to  do.  The  big 
change  for  instructors  is  that  they  may  not  be  giving  "stand-up"  presentations  anymore. 
However,  that  does  not  mean  that  all  they  do  is  grade  tests.  Every  instructor  should 
perform  at  least  the  following  five  functions  for  the  students  in  an  individualized  course: 

1.  Encouragement. 

2.  Problem  diagnosis. 

3.  Feedback. 

4.  Counseling. 

5.  Direct  instruction. 

This  looks  like  a  list  of  the  functions  instructors  perform  in  lock-step  courses,  which  is 
just  the  point.  Instructors  in  individualized  courses  do  all  these  things  on  an  individual 
basis.  You  might  call  giving  individual  attention  to  each  student  the  "bad  news."  The 
"good  news"  is  that  individualization  gives  the  instructor  time  to  do  these  things.  When 
you  are  not  spending  all  your  time  preparing  and  presenting  lectures,  you  have  time  to 
help  students. 

3.1  Encouragement  (Motivation) 

Studying  is  hard  work.  Learning  is  hard  work.  Putting  up  with  a  tough  instructor 
is  hard  work.  Therefore,  students  need  incentive.  They  need  some  motivation  to  keep  at 
it.  No  matter  how  interesting  your  individualized  course  is,  a  live,  breathing  human  being 
is  needed  to  bring  it  to  life.  tT he  more  alive  the  better.  If  the  instructor  seems  dull,  the 
subject  will  seem  dull,  too.) 

In  a  lock-step  system,  the  instructor  often  tries  to  bring  the  subject  to  life  in  a 
lecture.  Often,  however,  the  lectures  are  not  so  lively,  because  it  is  hard  to  know  what 
will  inspire  each  individual  in  the  class  at  the  same  time.  Individualization  can  make 
motivating  easier  because  it  is  easier  to  encourage  on  a  one-to-one  basis. 

The  instructor  personally  delivers  encouragement,  motivation,  incentive,  or 
whatever  you  call  it  in  an  individualized  course.  You  have  to  be  willing  to  dig  in  and  work 
with  the  student  if  you  are  going  to  individualize.  You  cannot  stay  on  the  platform. 

When  does  the  instructor  provide  motivation?  Since  there  is  usually  no  lecture, 
you  cannot  just  "plug"  motivation  into  the  lesson  plan.  However,  you  can  find  ways.  Some 
students  need  reassurance.  Others  need  a  goal  to  shoot  for.  Some  need  support  because 
they  are  weak  in  the  subject.  Others  need  a  verbal  shove  to  get  moving.  You  can  tell 
these  things.  You  get  to  know  what  different  students  need  by  working  with  them. 

In  an  individualized  course,  the  instructor  can  provide  some  words  of  encourage¬ 
ment: 


1.  Before  a  student  begins  work  for  the  day. 

2.  After  correcting  a  student's  work. 

3.  When  a  student  needs  an  explanation. 

4.  At  the  end  of  a  test  or  class  session. 
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These  are  only  a  few  of  the  obvious  times.  An  instructor  in  the  classroom  will  find  many 
more.  The  important  thing  is  to  avoid  the  notion  that  individualized  instruction  means  a 
"hands-off"  attitude  toward  motivating  students.  This  attitude  is  incorrect.  Do  not 
believe  instructors  who  say  they  will  not  have  anything  to  do  if  their  course  is 
individualized. 

3.2  Problem  Diagnosis 

Instructors  absolutely  must  diagnose  problems  that  students  are  having  with  the 
instruction.  Suppose  you  are  the  instructor.  If  one  of  your  students  fails  to  master  a 
lesson,  who  will  find  the  reasons  for  the  failure?  If  the  student  had  understood  the 
problem,  there  probably  would  not  have  been  a  problem,  which  leaves  the  instructor  to 
find  out  why  the  student  did  not  master  the  lesson.  Figuring  out  what  the  student  is  doing 
wrong  is  probably  the  biggest  single  part  of  the  instructor's  job  in  individualized 
instruction.  There  is  not  much  point  to  individualizing  unless  you  are  going  to  diagnose 
students'  work  individually. 

The  word  diagnose  means  more  than  just  approval  or  disapproval.  These  are  part 
of  providing  feedback,  which  we  will  discuss  next.  Right  now  we  are  talking  about 
figuring  out  what  the  students  are  doing  wrong.  To  figure  out  their  problem,  you  cannot 
just  look  at  a  test  score.  To  say,  "You  are  doing  terribly,"  does  not  help  much  either. 

First,  find  out  what  the  students  are  doing.  You  have  to  know  how  they  solved  a 
problem,  for  instance,  before  you  can  tell  them  where  they  went  wrong.  Do  not  start  by 
telling  them  how  to  do  it  right.  You  are  probably  better  off  letting  the  students  tell  you 
what  they  did. 

Example:  Suppose  the  subject  is  fire  support  coordination.  If  a  student  works  a 
problem  and  gets  the  wrong  answer,  get  him  or  her  to  walk  through  it  step-by-step.  Do 
not  correct;  just  listen  and  then  you  can  tell  the  student  why  he  or  she  got  the  wrong 
answer. 

Sometimes  the  problem  is  obvious  and  you  can  tell  the  student  at  once  how  to 
get  right  answers  instead  of  wrong  ones.  A  lot  of  the  time,  however,  you  have  to  do  some 
detective  work  and  figure  out  what  is  happening  inside  the  student's  head.  Is  it  confusion 
between  two  new  terms?  Is  it  misunderstanding  a  new  concept?  Is  it  a  wrong 
assumption?  Let  the  student  do  some  talking  before  jumping  in.  Remember  to  try  to 
figure  out  how  the  student  can  avoid  the  same  mistake  later.  If  the  student  makes  the 
same  mistake  again,  maybe  the  instructor's  diagnosis  was  incomplete. 

3.3  Feedback 


How  am  I  doing?  Am  I  doing  it  right?  These  questions  are  very  basic.  When  you 
are  trying  to  learn  something,  you  need  to  know  whether  or  not  you  are  catching  on. 
Another  critical  aspect  of  the  instructor's  job  in  an  individualized  course  is  to  answer 
these  questions  for  all  the  students  and  individualization  allows  the  instructor  more  time 
to  do  it.  (In  a  lecture,  you  cannot  tell  If  each  student  understands.)  Although  good 
instruction  materials  also  tell  students  how  they  are  doing,  they  are  never  enough. 
Because  individuals  react  to  instruction  differently,  you  cannot  predict  all  the  ways  in 
which  students  can  go  wrong. 

The  instructor  in  any  instruction  system— including  individualized  instruction— 
has  to  tell  students  whether  their  work  is  "on  or  off  target,"  and,  in  other  words,  steer  the 
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students  toward  perfect  performance.  Guide.  Coach.  The  more  "corrective  feedback*' 
that  students  get,  the  faster  they  will  get  where  they  are  going. 


The  classic  military  example  of  corrective  feedback  is  in  long-range  gunnery. 
Artillery  and  naval  gunfire  need  feedback  to  destroy  a  target.  The  forward  observer  (FO) 
or  naval  gunfire  spotter  is  the  "instructor"  who,  by  calling  in  corrections,  steers  the 
rounds  closer  and  closer. 

Long-range  gunfire  needs  feedback  to  be  effective;  students  need  feedback  to 
learn.  Without  feedback  from  the  instructor,  the  students  are  not  effective.  (They  will 
eventually  learn  because  they  can  get  feedback  from  the  instruction  materials,  just  as  the 
artillery  could  eventually  neutralize  a  target  without  an  FO— but  a  lot  of  energy  will  be 
wasted  if  there  is  no  guidance  along  the  way.) 

Part  of  the  instructor's  job  in  an  individualized  course  is  to  give  feedback— to 

coach,  to  guide,  and  to  steer.  This  is  one  more  reason  why  it  is  wrong  to  think  that 

individualization  means  "getting  rid  of  the  instructor."  The  best  time  for  the  instructor  to 
give  feedback  is  when  the  students  are  practicing.  To  make  practical  exercise  effective, 
the  students  need  feedback  about  their  performance.  The  instructor  should  tell  the 
students  what  is  wrong,  what  is  right,  and  how  to  improve. 

1.  Give  complete  feedback.  Good  feedback  is  complete.  It  consists  of  more 

than  just  right  or  wrong,  but  includes  diagnosis  too.  If  the  instructor  can  tell  the  students 

why  they  are  "off  target,"  that  is  all  the  better. 

2.  Give  feedback  immediately.  During  a  series  of  practice  trials,  give 
feedback  after  each  trial.  Do  not  wait  until  the  same  mistake  has  been  made  twice  to 
correct  it.  Would  an  FO  call  more  rounds  into  the  wrong  coordinates  before  he  gave  a 
correction? 

3.4  Counseling 

In  an  instruction  system,  there  will  be  "students  with  problems"  and  "problem 
students."  Students  with  problems  come  to  the  instructor  for  help  (or  the  instructor 
knows  they  need  help).  It  may  be  an  academic  or  personal  problem. 

Instructors  refer  serious  personal  problems  to  someone  with  professional 
training,  because  they  are  not  trained  professional  counselors.  Instructors  in 
individualized  courses  should  counsel  students  with  academic  problems,  just  as  they  would 
in  lock-step  courses. 

Problem  students  give  the  instructor  problems  (you  might  say  the  instructor  has 
problems  caused  by  the  students).  Maybe  the  students  are  absent  too  much,  late,  or  not 
trying.  The  same  problems  occur  in  individualized  instruction  as  in  lock-step  courses. 

Again,  individualization  allows  instructors  more  flexibility  in  conducting  a 
counseling  session.  It  does  not  have  to  be  after  duty  hours,  but  can  be  on  the  spot, 
because  the  instructor  will  not  be  taking  time  away  from  the  whole  class.  For  more 
information  on  this  role  of  the  instructor,  look  at  the  instructional  management  course 
lesson  on  counseling. 
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3.5 


Direct  Instruction 


Direct  instruction  is  communicating  content  to  the  students.  Although  most  of 
the  instruction  in  an  individualized  course  is  delivered  by  materials  and  not  by  an 
instructor,  there  are  many  things  the  materials  cannot  teach. 

In  a  course  teaching  communication,  for  example,  instructors  may  demonstrate 
some  procedure  or  model  some  task  so  students  can  see  how  to  do  it.  There  are  many 
opportunities  like  this  in  individualized  courses. 

Remember  that  direct  instruction  does  not  have  to  be  given  to  the  whole  class  at 
once.  You  can  individualize  a  course  and  still  give  one-to-one  demonstrations.  The 
difference  is  that  in  the  individualized  course  you  have  more  chance  to  tell  whether  the 
student  understands  what  you  are  demonstrating. 

3.6  Summary 


An  instructor  in  an  individualized  course  must  provide  encouragement,  problem 
diagnosis,  feedback,  counseling,  and  some  direct  instruction  to  all  students  individually. 
All  instructors  should  perform  these  duties  in  all  courses;  however,  in  individualized 
courses,  the  emphasis  is  more  on  the  first  four  duties  and  less  on  the  fifth,  direct 
instruction.  That  is  part  of  the  point  of  individualized  instruction.  Instructional  materials 
provide  most  of  the  direct  instruction,  which  gives  the  instructor  more  time  to  motivate, 
diagnose  learning  problems,  and  provide  feedback  and  counseling.  Human  beings  generally 
do  those  things  better  than  do  videotapes  or  texts. 

4.0  DESIGNING  YOUR  INDIVIDUALIZED  SYSTEM 

To  put  an  individualized  system  together,  you  need  to  decide  which  factors  to 
vary  in  the  system.  In  Chapter  2,  you  saw  bow  different  systems  vary  the  content,  time, 
method,  and  achievement  level.  You  also  need  to  think  about  how  to  include  each  of  the 
basic  components  that  belong  in  any  instruction  system:  presentation,  student  response, 
feecoack,  and  evaluation.  You  have  already  learned  about  these  in  your  instructional 
management  course.  As  including  all  of  the  basic  components  is  even  more  important  in 
an  individualized  system,  this  chapter  will  begin  by  reviewing  them. 

4.1  Instruction  System  Components 

The  simplest  instruction  system  has  one  part:  presentation.  An  example  of  this 
system  is: 


FILM. 

Have  you  ever  attended  a  mandatory  class,  perhaps  on  the  Geneva  Conventions?  Often  it 
consists  only  of  showing  a  film. 

A  slightly  more  advanced  instruction  system  has  two  parts:  presentation 
and  evaluation.  For  example, 


LECTURE  ♦  TEST. 
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The  class  in  American  history  described  in  Chapter  1  is  an  example  of  such  a  system. 
There  are  just  two  parts:  The  instructor  talks  and  the  students  respond  on  a  test.  Most 
instruction  systems  have  more  parts  than  these.  You  should  include  all  four  parts  in  any 
instruction  system  you  build  (even  a  lock-step  system);  however,  they  are  even  more 
important  with  individualized  instruction.  With  lock-step  system,  you  can  "cover"  your 
mistakes  on  the  spot.  If  you  leave  anything  out  of  an  individualized  system,  look  out! 
Once  again,  here  they  are: 

1.  Presentation. 

2.  Student  response. 

3.  Feedback. 

4.  Evaluation. 

Obviously,  the  goal  of  the  instruction  system  is  to  get  the  content  over  to  the 
students.  In  the  past,  many  instructors  did  not  think  about  anything  but  packaging  the 
content  in  the  best  lecture  they  could  give.  They  took  the  "fire-and-forget"  approach: 
Shoot  the  content  at  the  students  and  hope  you  hit  as  many  as  possible.  That  is  not  the 
most  effective  system  for  covering  the  target. 

The  other  parts  of  your  instruction  system  are  important  too.  Each  serves  a 
purpose  in  getting  the  students  to  learn.  We  will  address  them  one  by  one. 

4.1.1  Presentation 

A  presentation  is  any  input  to  the  students,  including  a  statement  of  the  learning 
objective,  explanations,  modeling  of  performance,  examples  of  the  desired  product— any 
transfer  of  information.  Presentations  may  be  made  by  instructional  materials  (texts, 
videotape  program,  film,  tape/slide  program),  by  instructors,  and  by  other  students. 

Remember  that  this  is  only  one  part  of  the  system.  It  is  not  enough  just  to  put 
forth  the  subject  to  the  students.  You  have  to  think  about  how  you  are  going  to  make  sure 
they  learn  the  content.  (If  you  think  that  is  the  students'  problem,  you  are  wrong.  That  is 
like  saying  that  a  basketball  coach  has  no  responsibility  for  the  team's  record.) 

4.1.2  Student  Response 


Output  from  the  students  includes  activities  such  as  asking  the  instructor 
questions,  answering  questions,  rehearsing  behavior  to  be  learned,  and  completing  practice 
exercises.  This  item  is  often  forgotten  by  instructors,  who  tend  to  fill  up  an  entire  lesson 
with  their  presentation.  This  leaves  no  room  for  active  response  by  the  students. 

Students  learn  better  when  they  respond  actively  to  instruction  instead  of  just 
listening.  Practice,  or  some  other  form  of  response,  keeps  them  alert  and  helps  them 
remember  what  they  have  learned.  A  good  example  is  the  "conference  group"  method  of 
instruction.  When  students  are  actively  engaged  in  responding  to  a  problem,  they  can 
learn  more  than  if  they  are  just  "talked  to." 

Questions  are  the  simplest  form  of  student  response.  Even  a  simple  instruction 
system  (like  lecture  plus  test)  allows  students  to  respond  to  the  lecture.  A  more  advanced 
form  of  student  response  is  practice,  as  in  conference  groups.  With  individualized 
instruction,  it  is  important  to  allow  lots  of  practice.  That  is  when  the  instructor  can  see 
how  the  students  are  doing. 


22 


4.1.3  Feedback 


Practice  must  be  followed  by  feedback.  Any  student  response  must  be  followed 
by  feedback.  In  a  lecture,  feedback  is  simple.  If  students  ask  questions,  the  instructor 
just  answers  them- -that  is  feedback. 

In  your  individualized  system,  you  will  have  to  plan  for  feedback,  since  it  does 
not  just  happen.  In  Chapter  3,  you  read  that  giving  feedback  was  part  of  the  instructor's 
job.  There  are  also  several  other  ways  to  give  feedback. 

The  most  obvious  one  is  answers  to  practice  exercises.  In  the  workbook/tutorial 
system,  the  answers  are  at  the  back  of  the  workbooks.  In  a  tape/ slide  lesson,  you  can 
include  feedback  on  practice  questions  right  on  a  slide  or  on  the  audiotape.  For  example, 
after  the  students  have  been  given  the  question,  the  next  slide  can  give  the  answer. 

Another  time  you  will  give  feedback  is  after  a  quiz  or  test.  In  your 
individualized  system,  you  can  critique  the  performances  of  all  students  separately. 
Often,  an  individualized  system  requires  students  to  master  a  unit  before  proceeding,  so 
the  students  may  have  to  take  more  than  one  test  on  the  same  subject.  Feedback  on  test 
performance  is  especially  important.  You  have  to  tell  the  students  what  they  are  doing 
wrong  so  they  can  improve. 

Feedback  should  come  as  soon  as  possible.  After  the  students  have  finished  a 
practical  exercise  or  a  test,  you  should  give  the  feedback  right  away.  This  means  planning 
your  individualized  system  so  that  feedback  will  occur  at  the  right  time. 

4.1.4  Evaluation 


Evaluation  is  a  step  you  probably  will  not  leave  out.  Almost  everyone  includes  a 
test  in  instruction  and  individualized  instruction  usually  requires  a  lot  more  testing  than 
lock-step  instruction.  The  reason  is  simple.  In  lock-step  instruction,  the  student  often 
just  takes  a  final  test,  such  as  an  American  history  exam.  If  the  students  pass  or  not,  the 
instruction  system  often  does  nothing  about  it. 

Individualization  provides  the  students  and  instructor  with  opportunities  to 
review  the  test  and  make  sure  the  students  understand  what  is  being  taught.  If  students 
do  not  pass,  the  system  usually  has  more  instruction  and  more  tests  available.  Tests  are 
used  to  control  the  student  progress  through  the  program.  You  will  probably  have  a  lot 
more  tests  than  you  would  in  lock-step.  You  will  probably  also  need  three  or  more  forms 
of  each  test  with  different  items.  You  may  not  want  to  give  the  same  questions  again  to 
a  students  who  did  not  master  a  lesson. 

How  to  construct  tests  is  covered  in  Chapter  8  of  this  handbook,  in  Marine  Corps 
Order  1553.1  (Phase  III),  in  Interservice  Procedures  for  Instructional  Systems  Development 
(Phase  II)  (NAVEDTRA  106A),  in  the  instructional  management  course  lesson  on  test 
construction,  and  in  the  Handbook  for  Testing  in  Navy  Schools  (Ellis  it  Wulfeck,  1982). 

4.1.5  Summary  of  Instruction  System  Components 

We  have  reviewed  four  "building  blocks"  that  you  should  put  in  your  individ¬ 
ualized  instruction  system: 

1.  Presentation. 

2.  Student  response. 
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3.  Feedback. 

4.  Evaluation. 

In  fact,  these  four  components  belong  in  any  instruction  system.  In  an  individualized 
system,  none  of  these  parts  can  be  left  to  chance.  You  have  to  plan  how  you  will 
incorporate  each  into  the  system.  This  makes  an  individualized  system  harder  to  plan 
than  a  lock-step  system.  Also,  the  better  you  plan  your  system,  the  better  it  will  work. 

We  will  now  take  an  individualized  system  that  you  are  familiar  with  and  see  how 
each  of  these  parts  is  included.  We  will  use  the  workbook/tutorial  system  at  IMS  as  an 
example.  Notice  that  tests  serve  both  as  student  responses  and  evaluations. 

1.  Presentations.  The  student  reads  instructional  workbooks. 

2.  Student  response.  The  student: 

a.  Answers  questions  in  the  workbooks. 

b.  Develops  products  (lesson  plans,  tests,  objectives,  15-minute  presen¬ 
tations,  etc). 

c.  Asks  questions. 

d.  Answers  instructor's  questions. 

e.  Takes  tests. 

3.  Feedback.  The  student: 

a.  Reads  the  answers  to  the  questions  in  the  workbook. 

b.  Is  told  the  good  and  bad  points  of  his  or  her  products. 

c.  Is  told  what  he  or  she  got  right/wrong  on  the  tests. 

d.  Receives  answers  to  his  or  her  questions. 

4.  Evaluation.  The  student  takes  lesson  tests. 

Study  the  ways  these  parts  of  an  instruction  system  have  been  included  in  the 
workbook/tutorial  system  at  IMS.  Now  think  about  how  you  would  include  them  in  a 
course  you  might  design. 

4.2  Your  System's  Program 

Figure  4-1  shows  the  basic  program  you  should  have  in  mind  to  build  your 
individualized  system.  The  actual  program  you  need  to  build  is  more  complicated.  You 
need  to  build  in  repeated  responses  and  feedback  within  each  lesson  or  module,  as  shown 
in  Figure  4-2. 

In  a  real-life  example  of  repeated  student  responses  and  feedback  from  the  IMS 
lesson  on  developing  objectives,  students  are  learning  how  to  write  the  behavior  portion  of 
a  learning  objective.  Figure  4-3  shows  the  first  student  activity.  The  first  feedback, 
which  is  answer  key  02,  is  shown  in  Figure  4-4.  Figure  4-5  presents  a  second  student 
activity;  and  Figure  4-6,  the  feedback  (key  05). 
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Presentation.  Here  you  in- 
troduce  a  portion  of  the 
content,  such  as  one  lesson 
or  module. 

dmdcnt  response.  Here  you 
have  the  student  practice  or 
respond  to  the  presentation 
in  some  way. 

Feedback.  Here  you  provide 
answers  to  the  questions  the 
students  have  completed  or 
you  provide  models  of  the 
correct  response  for 
practice  exercises. 

Evaluation.  Here  you  find 
out  whether  the  student  has 
mastered  the  lesson. 


Figure  4-1.  Basic  program  for  instruction. 


Practice 


More 

Practice 


More 

Practice 


Figure  4-2.  Repeated  response  and  feedback 


For  each  learning  objective  below,  underline 
the  behavior.  (See  Key  It 2  at  the  end  of  the 
lesson  for  answers.) 

1.  Perform  operator  maintenance  on  on  AN/PRC-77. 

2.  State  the  function  of  each  part  of  a  carburetor. 

3.  Run  the  hundred-yard  dash  on  a  dry  track  within 
fourteen  seconds. 

4.  List  the  five  planning  considerations  for  conducting  an 
amphibious  operation,  without  the  aid  of  references. 

5.  Civen  a  rough  draft,  typewriter,  and  paper,  type  a  letter 
at  a  minimum  speed  of  40  words  per  minute. 


Figure  4-3.  Example  of  initial  student  response. 


1.  Perform  operator  maintenance  on  an  AN/PRC-77. 

2.  State  the  function  of  each  part  of  a  carburetor. 

3.  Run  the  hundred-yard  dash  on  a  dry  track  within 
fourteen  seconds. 

4.  List  the  five  planning  considerations  for  conducting  an 
amphibious  operation,  without  the  aid  of  references. 

5.  Given  a  rough  draft,  typewriter,  and  paper,  type  a  letter 
at  a  minimum  speed  of  40  words  per  minute. 


Figure  4-4.  Feedback  to  initial  student  response. 
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PRACTICE:  In  the  following  objectives, 

1.  Underline  the  BEHAVIOR. 

2.  Check  whether  the  action  is  observable  or 
unobservable.  (See  Key  #5.) 

1.  Given  the  values  for  voltage,  current,  or  resistance  of  an 
electrical  current,  calculate  the  unknown  values,  using  Ohm's  law. 

_  observable  unobservable 

2.  Given  the  appropriate  TM,  be  familiar  with  the  lubricating 
procedures  for  the  Ml  6  rifle* 

_  observable  unobservable 

3.  Without  aid  of  references,  know  the  major  steps  of  logistic 
planning  in  accordance  with  FMFM  4-1. 

_  observable  unobservable 

4.  Define  in  writing  the  key  financial  management  terminology 
utilized  within  the  Marine  Corps. 

observable  _  unobservable 


Figure  4-5.  Example  of  second  student  response. 


1.  Given  Che  values  for  voltage,  current,  or  reslstence  of  an 

electrical  current,  calculate  the  unknown  veluee,  using  Ohm’s  lew, 

observable  _  unobserveble 


2.  Given  the  appropriate  TM,  be  femlllar  with  the  lubrlceting 
procedures  for  the  M16  rifle. 

_  observable  unobservable 


3. 


Without  eld  of  references,  know  the  major  eteps  of  logistic 
planning  in  eccordence  with  FMFM  4-1. 


observeble 


J- 


unobserveble 


4. 


Define  In  writing  the  key  flnanclel  manegement  terminology 
utilized  within  the  Marine  Corpe. 


obeervable 


unobserveble 


Figure  4-6.  Feedback  to  second  student  response. 
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Figure  4-7  is  a  more  complete  diagram  of  an  individualized  instruction  system 
showing  how  to  include  repeated  student  responses  and  feedback. 


Figure  4-7.  Complete  individualized  instruction  system. 


4.3  Providing  Presentations 

Figure  4-8  shows  the  presentation  component  of  an  individualized  instruction 
system.  How  to  present  the  content  of  your  course  is  usually  the  first  question  you  will 
answer  in  individualizing.  Will  you  use  one  method  of  presentation  or  several?  Here  are 
some  of  the  possibilities  (more  are  listed  in  the  Interservice  Procedures  for  Instructional 
Systems  Development,  Phase  III  (NAVEDTRA  106 A),  Appendix  A): 

1.  Printed  text  or  workbook. 

2.  Videotape. 

3.  Tape/slide. 

4.  Motion  picture. 

5.  Closed-circuit  television. 

6.  Computer. 

7.  Lecture/demonstration. 

8.  Conference  group. 

9.  Teleconference. 

10.  Programmed  text. 


28 


Figure  4-8.  Presentation  component. 


Of  course,  the  decision  about  what  presentation  mode(s)  to  use  must  be  based  on 
what  you  are  teaching--on  the  objectives  of  instruction.  A  presentation  mode  such  as  a 
demonstration,  for  instance,  might  be  good  for  teaching  how  to  conduct  a  river  crossing. 
For  a  conceptual  subject  such  as  national  security  policy,  however,  a  physical  demon¬ 
stration  is  not  really  needed. 

For  some  objectives  you  will  have  to  represent  motion  or  color.  For  others,  you 
may  only  need  words.  The  procedures  for  selecting  methods  of  presenting  content  are 
spelled  out  in  Marine  Corps  Order  1553.1,  in  the  Interservice  Procedures  for  Instructional 
Systems  Development  (NAVEDTRA  106A),  and  in  the  IMS  lesson  on  selecting  instructional 
methods  and  media. 

Suppose,  for  example,  you  determined  that  a  tape/slide  presentation  would  be  best 
for  presenting  content,  at  least  at  first.  In  that  case,  your  program  would  begin  to  look 
like  this: 


Student  views  tape/slide 
for  first  part  of  lesson 

- 1 - 

4.4  Providing  for  Student  Response 

Figure  4-9  shows  the  student  response  component.  Some  commonly  used  ways  for 
the  students  to  respond  to  the  presentation  are  to: 

1.  Provide  study  questions  such  as  flll-in-the-biank  and  true/false. 

2.  Provide  a  short  multiple-choice  quiz. 

3.  Describe  a  problem  or  case  history  and  require  the  student  to  give  a  solution 
or  opinion.  Example:  "Fire  mission  called  to  3/11  on  840810  by  artillery  forward  observer 
with  2/7.  What  should  the  fire  support  coordinator  do?" 
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Figure  4-9.  Student  response  component. 


4.  Provide  drill  or  practice  exercise  in  which  the  student  does  the  same  thing 
many  times  to  build  up  speed  and  accuracy.  Example:  Require  the  student  to  identify  a 
succession  of  threat  and  friendly  armored  vehicles. 

5.  Require  the  student  to  create  a  product.  Example:  Write  an  objective,  fill 
out  a  form,  or  develop  a  scheme  or  maneuver. 

The  kind  of  student  activity  you  should  provide  depends  in  part  on  what  you  are 
teaching.  If  you  are  teaching  concepts,  give  lots  of  examples.  If  you  are  teaching  any 
recognition  skill,  give  lots  of  drill-type  practice.  The  kinds  of  student  activities  needed  to 
teach  different  objectives  are  spelled  out  in  the  Interservice  Procedures  for  Instructional 
Systems  Development  (NAVEDTRA  106 A)  and  in  the  Instructional  Quality  Inventory:  11. 
User's  Manual  (EllisTWulfeck,  &  Fredericks,  1979). 

Suppose  you  determined  that  the  student  should  respond  to  the  presentation  in  two 
ways:  (1)  by  taking  a  short  multiple-choice  quiz  and  (2)  by  creating  a  written  product.  In 
that  case,  your  system  would  look  like  Figure  4-10. 

4.3  Providing  Feedback 

You  may  have  noticed  a  problem  in  Figure  4-10.  Since  there  are  two  kinds  of 
student  response,  there  should  be  two  kinds  of  feedback.  The  response-feedback 
combination  should  really  look  like  that  shown  in  Figure  4-11. 

However,  you  could  also  use  the  results  of  the  first  response  to  determine  whether 
or  not  the  student  needs  more  practice,  as  shown  In  Figure  4-12.  In  this  example,  a 
student  who  scored  90  percent  on  the  quiz  would  continue.  A  student  who  scored  below  90 
percent  would  stop  and  create  a  written  product.  Building  alternate  pathways  like  this 
into  your  Instruction  program  Is  called  branching.  By  providing  two  different  branches, 
faster  students  can  keep  moving  and  students  who  need  more  practice  can  get  more 
practice. 


30 


Figure  4-10.  Two  examples  of  student  response. 
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Figure  4-12.  Example  of  branching. 


There  are  three  basic  methods  for  providing  feedback  to  your  students: 

1.  Answer  keys. 

2.  "Model"  answers. 

3.  Instructor  critique. 

You  can  use  answer  keys  when  the  students  have  completed  a  quiz  or  fill-in-the-blank 
exercise.  A  model  answer  is  like  a  "school  solution."  You  can  provide  a  model  answer  if 
the  students  have  given  an  opinion  or  solution  to  a  problem.  The  instructor  critique,  of 
course,  gives  you  the  best  way  to  see  how  the  students  are  doing.  It  gives  the  instructor  a 
chance  to  diagnose  any  problems  they  are  having  and  an  opportunity  to  motivate  and 
encourage  them. 

On  the  other  hand,  since  the  instructor  does  not  have  the  time  to  give  feedback 
to  every  student  every  5  minutes,  a  lot  of  feedback  must  come  from  the  instructional 
materials.  Probably  the  best  solution  is  to  combine  the  feedback  from  the  materials  and 
the  instructor.  That  way  the  instructor  can  also  make  sure  his  or  her  instruction 
materials  are  doing  the  job. 

In  the  example  we  are  using,  suppose  that  you  decide  to  use  answer  keys  for 
feedback  on  the  quizzes  and  instructor  critique  as  feedback  on  the  written  product.  Your 
system  would  look  like  that  shown  in  Figure  4-13. 
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Figure  4- 13.  Two  types  of  feedback. 


4.6  Evaluation 

How  will  you  know  if  the  student  has  mastered  the  lesson  objective(s)?  This  is 
the  critical  point  in  a  performance-based  individualized  instruction  system.  If  you  do  not 
evaluate  carefully,  your  system  will  not  work.  There  are  three  decisions  you  have  to 
make  in  evaluating  students. 

1.  What  to  cover.  You  or  someone  else  must  spell  out  exactly  what  students 
are  supposed  to  learn  before  you  can  plan  your  tests.  This  should  be  done  in  well-written 
instructional  objectives.  If  it  is  not,  you  will  have  to  define  the  desired  student 
performance  before  you  develop  the  tests. 

2.  How  to  sample.  You  may  not  be  able  to  cover  the  entire  performance. 
Suppose  you  want  the  students  to  be  able  to  write  a  good  plan  for  tasking  artillery.  How 
much  of  such  a  plan  should  they  write?  Should  they  write  several  plans?  These  are 
questions  of  sampling.  Since  you  cannot  always  have  students  do  everything,  you  may 
have  to  sample  from  all  the  possibilities.  Ask  yourself,  "What  are  the  critical  points  in 
the  task?"  Then,  concentrate  on  those. 

3.  What  kind  of  test  to  use?  The  test  you  use--paper-and-pencil  knowledge, 
hands-on  performance,  multiple-choice,  or  free  response— depends  on  what  you  are  trying 
to  measure.  You  should  consider  the  different  possibilities  before  you  start.  The  pros  and 
cons  of  different  types  of  tests  are  discussed  later  in  this  handbook.  You  can  also  learn 
more  about  this  decision  in  the  Instructional  Quality  Inventory:  n.  User’s  Manual  (Ellis, 
Wulfeck,  Sc  Fredericks,  1979)  and  in  the  Handbook  for  Testing  in  Navy  Schools  (Ellis  Sc 
Wulfeck,  1982). 
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For  our  example,  suppose  you  decide  to  use  a  hands-on  performance  test, 
producing  a  written  product.  We  call  this  a  performance  test,  even  though  paper  and 
pencil  are  used.  Many  tasks  (such  as  developing  a  lesson  plan)  are  performed  with  paper 
and  pencil.  At  the  staff  officer  level,  for  instance,  many  tasks  involve  planning.  There  is 
not  always  a  contradiction  between  "performance"  and  "paper  and  pencil." 

Figure  4-14  shows  your  system  now,  using  a  hands-on  performance  test  to 
evaluate  performance. 


Figure  4-14.  Evaluation  component. 


4.7  Additional  Presentation/Response/Feedback/Test 

Suppose  the  student  passes  the  test.  The  instructor  can  review  the  test  with  the 
student  and  the  student  can  move  on  to  the  next  lesson  or  objective  or,  if  this  is  the  last 
lesson,  to  graduation! 

Suppose,  however,  the  student  does  not  pass  the  test.  What  will  your  system  do? 
We  will  see  that  this  is  where  individualiztion  really  pays  off. 

Suppose  you  decided  to  use  an  alternate  presentation  method  for  students  who 
fail  to  master  the  lesson  test.  Perhaps  you  can  use  a  good  programmed  text  that  covers 
the  same  objectives.  You  could  set  up  your  individualized  system  so  that  students  who 
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fail  the  lesson  test  study  the  programmed  text.  After  that,  they  would  retake  the  lesson 
test,  as  shown  in  Figure  4-15. 


© 


ANSWERS  TO  QUIZ 


ADOITIONAL  a 
PRESENTATION  1 
STUDENT  RESPONSE, 
AND  FEEOBACK 


ADDITIONAL 

EVALUATION 


Figure  4-15.  Additional  instruction  after  failure  to  pass  test. 


If  the  student  does  not  pass  the  test  at  this  point,  here  are  some  of  the 
possibilities  that  you  could  build  into  your  system: 

1.  Restudy  primary  presentation  (in  this  example,  tape/ slide). 

2.  Restudy  secondary  presentation  (in  this  example,  programmed  text). 

3.  Schedule  a  one-to-one  tutorial  with  instructor. 

4.  Provide  a  new  method  of  instruction. 

5.  Schedule  a  one-to-one  tutorial  with  another  student. 

Suppose  that  you  provide  an  instructor  tutoring  session  after  the  retest.  At  that 
time,  the  instructor  could  (1)  review  tire  student’s  test,  (2)  diagnose  problems,  and  (3) 
prescribe  additional  restudy,  as  shown  in  Figure  4-16.  In  this  example,  the  instructor 
assigns  additional  work  to  the  student  after  the  test  review  and  diagnosis.  After  finishing 
this  extra  study,  the  student  takes  the  lesson  test  for  the  third  time. 
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If  the  student  still  does  not  pass  at  this  point,  what  will  the  system  do?  Some 
possibilities  you  could  consider  are: 

1 .  Another  instructor  tutorial. 

2.  Restudy  presentations. 

3.  Study  new  presentation  {a  little  late  for  this,  probably). 

4.  Move  on  to  next  lesson. 

5.  Instructor  decision. 
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Remember,  these  are  just  a  few  possibilities.  You  can  consider  many  other  actions,  too. 
The  last  item,  instructor  decision,  means  that  the  instructor  has  to  decide  what  action  to 
take.  Be  careful  about  putting  a  step  like  this  into  your  system,  because  the  instructor 
might  be  "swamped"  with  a  lot  of  on-the-spot  decisions.  If,  however,  you  think  that  few 
students  will  fail  the  lesson  test  three  times,  it  may  not  be  a  problem. 


Summary 


You  have  just  seen  an  example  of  how  one  individualized  system  could  be  built. 
To  build  your  system,  you  should  concentrate  on  the  following  general  questions  that  we 
had  to  deal  with  along  the  way. 


1.  How  will  you  present  instruction  initially? 

2.  How  will  you  provide  for  the  student's  response? 

3.  How  will  you  provide  feedback  to  the  student's  response? 

4.  How  will  you  evaluate  the  student? 

5.  If  the  student  fails  the  first  evaluation,  what  will  you  provide  in  the  way  of: 

a.  Additional  presentations? 

b.  Additional  student  response? 

c.  Additional  feedback? 

d.  Additional  evaluation? 

6.  What  will  the  system  do  if  the  student  "never"  passes  the  test? 


After  you  have  answered  each  one  of  these  questions,  you  will  be  on  your  way  to  building 
a  good  individualized  system.  If  you  do  not  cover  each  of  these  questions  ahead  of  time, 
your  system  may  fall  apart  when  students  begin  to  go  through  it. 

5.0  QUALITY  CONTROL  AND  MANAGEMENT 

Just  to  include  presentations,  feedback,  etc.,  in  your  individualized  system  is  not 
enough.  Their  quality  is  also  important  no  matter  if  you  convert  an  existing  course  or 
develop  a  new  course.  This  chapter  gives  you  guidance  on  how  to  control  the  quality  of 
your  system  and  how  to  set  up  a  student  management  system. 

In  the  last  chapter,  you  saw  how  to  build  an  individualized  instruction  system 
with  presentations,  student  activities,  feedback,  and  evaluation.  That  raises  some  big 
questions.  How  will  you  know  what  part  of  instruction  the  student  is  in?  Is  the  student 
ready  for  a  test?  What  is  the  student  supposed  to  do  next?  How  long  has  the  student  been 
working  on  a  lesson?  Unless  you  have  quick  answers  to  those  questions,  you  will  get 
confused.  Your  management  system  will  be  the  key  to  keeping  track  of  your  students  and 
their  progress. 

5.1  Quality  Control 


Everything  in  this  section  applies  to  both  individualized  and  lock-step  instruc¬ 
tion.  In  either,  the  quality  of  the  course  is  only  as  high  as  the  quality  of  its  parts. 

The  Navy  Personnel  Research  and  Development  Center  (NAVPERSRANDCEN) 
has  developed  guidelines  you  can  use  to  control  the  quality  of  your  instruction  system. 
Detailed  instructions  for  quality  control  of  objectives,  presentations,  student  response, 
feedback,  and  evaluation  are  in  the  Instructional  Quality  Inventory.  II.  User's  Manual 
(Ellis,  Wulfeck,  &  Fredricks,  1979).  This  section  is  based  on  the  instructional  quality 
inventory  (IQI)  procedures.  We  will  look  at  objectives  first. 
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5.1.1 


Are  Your  Objectives  Correctly  Stated? 


An  objective  serves  two  purposes.  It  tells  the  students  what  they  are  supposed  to 
learn  and  it  tells  the  instructors  what  they  are  supposed  to  teach. 

Telling  students  what  to  learn  is  a  powerful  tool — maybe  too  powerful.  If 
students  know  what  to  learn,  maybe  they  will  learn  it  alone  without  the  instructor.  Some 
instructors  see  this  as  a  threat.  They  do  not  want  to  tell  students  what  to  learn,  because 
they  do  not  want  to  lose  control  of  the  class.  Keeping  students  in  the  dark  is  a  good  way 
to  keep  control,  but  it  does  not  help  them  to  learn. 

Telling  instructors  what  to  teach  is  a  powerful  tool,  too.  Stating  objectives  of 
instruction  helps  keep  the  instructor  "on  track,"  which  is  important  no  matter  what 
method  the  instructor  is  using  (lesson  plan,  self-paced  lesson,  video  script,  or  whatever). 

A  big  problem  with  objectives  is  that  people  think  they  are  easy  to  state. 
However,  if  you  have  ever  tried  to  define  what  your  students  should  be  able  to  do,  you 
know  that  developing  objectives  is  difficult;  sometimes  it  takes  many  attempts  to  be  able 
to  state  what  you  think  is  obvious.  Developing  objectives  is  taught  in  Marine  Corps  Order 
1553.1,  in  Interservice  Procedures  for  Instructional  Systems  Development  (NAVEDTRA 
106 A),  and  in  the  IMS  lesson  on  developing  objectives. 

Every  objective  should  include  a  behavior  (action),  conditions,  and  a  standard.  If 
any  one  of  these  is  missing  for  objectives  in  your  course,  you  will  have  to  fix  them.  In  the 
following  objective,  the  conditions  are  missing:  "At  the  end  of  instruction,  debug  a 
computer  program  within  4  hours."  "At  the  end  of  instruction"  is  not  condition.  It  does 
not  tell  the  student  what  kind  of  program  will  be  debugged  nor  what  references  to  use.  It 
does  not  tell  the  instructor  enough  to  develop  the  instruction.  You  would  have  to  rewrite 
the  objective  before  developing  your  instruction. 

What  do  you  think  of  the  following  objective?  "Without  the  aid  of  references, 
describe  the  philosophy  of  a  force  of  combined  arms."  It  has  no  standard.  How  will  you 
know  whether  the  student  has  succeeded?  How  "good'’  does  the  description  have  to  be? 
How  much  and  what  level  of  detail  are  required?  As  the  objective  does  not  say,  you  would 
have  to  rewrite  it. 

When  you  individualize  a  course,  make  sure  that  each  objective  has  the  three 
essential  parts:  action,  conditions,  and  standard.  Also  make  sure  the  action  is  observable 
and  measurable. 

5.1.2  Are  Your  Presentations  Effective? 

The  IQI  gives  the  following  three  rules  for  making  your  presentations  effective: 

1.  Separate  important  points  from  the  rest  of  the  text  (or  lecture,  or 
audiovisual  lesson). 

2.  Identify  important  points  as  such. 

3.  Provide  some  "help"  to  remember  the  point. 

Let  us  see  how  these  rules  could  be  put  into  practice. 

1.  Separate  important  points.  There  are  lots  of  ways  to  separate  important 
points.  We  just  indented  the  three  IQI  rules.  Besides  indenting,  there  are  other  ways  to 
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separate  important  points  such  as  use  color,  use  a  box,  change  to  a  different  TYPE  FACE, 
or  put  a  special  point  on  a  completely  different  page.  In  an  audiovisual  lesson,  you  can 
pause  before  giving  the  statement  to  catch  the  listener's  attention. 

2.  Identify  important  points  as  such.  The  IQI  lists  the  following  labels  for  use 
in  identifying  important  points  you  want  the  students  to  remember: 

a.  Key  point. 

b.  Rule. 

c.  Definition. 

d.  Procedure  for _ . 

e.  Principle  of _ . 

-  f.  Main  idea. 

You  can  think  of  other  labels  that  match  the  content  of  your  own  course.  Here  is  an 
example  of  an  important  statement  separated  and  labeled: 


KEY  POINT.  An  objective 
has  three  parts:  (1)  action, 

(2)  conditions,  and  (3)  standard. 


3.  Provide  help.  When  you  present  an  important  point  that  the  student  must 
memorize,  include  something  to  help  the  student  understand  and  remember  it.  Here  are 
some  suggestions  from  the  IQI: 

a.  Provide  a  memory  trick  (mnemonic). 

b.  Give  an  example. 

c.  Explain  why  the  point  is  important. 

d.  Relate  the  point  to  something  else. 

e.  Explain  some  of  its  terms. 

f .  Illustrate  the  point  (symbol,  table,  flowchart,  etc.). 

For  example,  a  common  memory  trick  or  mnemonic  is  used  to  remember  the 
lines  of  a  staff  in  writing  music:  "Every  Good  Boy  Does  Fine"  (EGBDF).  Another  memory 
trick  is  used  in  biology  to  remember  the  divisions  of  the  natural  world:  "King  Phillip, 
Come  Out  For  Gosh  Sakes"  (Kingdom,  Phylum,  Class,  Order,  Family,  Genus,  Species).  In 
the  military,  many  memory  tricks  are  used  to  remember  concepts  and  principles.  If  you 
are  presenting  a  new  concept,  you  could  make  up  your  own  mnemonic  to  make  it  easier  to 
remember. 

Are  your  presentations  of  content  effective?  Remember  these  three  ways  to 
make  them  adequate. 

1.  Separate  important  points. 

2.  Identify  important  points. 

3.  Provide  help. 

5.1.3  Are  Your  Examples  Adequate? 

Adequate  presentations  need  adequate  examples.  When  you  give  examples,  the 
IQI  says  to  follow  these  rules: 

1 .  Examples  must  be  separated  and  identified. 
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2.  Examples  must  include  some  type  of  help. 

3.  Examples  should  range  from  easy  to  hard. 

4.  Examples  should  represent  the  job  the  student  will  do  after  training. 

5.  There  should  be  enough  examples  to  cover  the  content  area  adequately. 

6.  Examples  should  show  clearly  why  common  errors  are  wrong. 

As  you  can  see,  the  first  two  rules  for  examples  are  the  same  as  the  rules  for 
presentations.  Separate  and  identify  them.  Provide  some  help  in  understanding  them. 

Help  (Rule  2)  is  included  in  different  ways  for  different  situations.  Read  the  1QI 
manual  to  see  how  to  give  help  for  categories,  procedures,  and  principles  and  for  more 
complete  guidance. 


Are  Your  Practice  Items  A  dec 


Active  student  response- -usually  practice- -should  be  part  of  your  instruction 
system.  To  control  the  quality  of  your  system,  the  IQI  gives  these  rules  for  practice 
items: 


1.  The  practice  section  must  be  separate  and  identified. 

2.  The  practice  items  must  be  free  of  hints  that  will  not  be  present  in  the  test 
or  on  the  job. 

3.  The  format  of  the  practice  and  test  items  shouid  be  the  same. 

4.  The  practice  items  should  range  from  easy  to  hard. 

5.  The  practice  items  should  be  typical  of  the  job  to  be  performed  after 

training. 

6.  The  practice  items  should  include  the  opportunity  for  common  errors. 

7.  The  practice  item  must  be  followed  by  feedback. 

These  rules  are  self-explanatory;  read  the  1Q1  manual  to  learn  more.  The  last  item  listed, 
feedback,  brings  us  to  our  next  quality-control  question. 

5.1.5  Are  Your  Feedback  Items  Adequate? 

Sometimes  you  give  the  answers  (feedback)  to  practice  questions  in  the  back  of  a 
test,  workbook,  or  study  guide.  You  can  also  give  them  right  along  with  the  presentation. 
When  you  do  this,  the  IQI  says  to  observe  these  rules: 

1.  The  feedback  must  also  be  separated  and  identified  for  each  practice  item. 

2.  The  feedback  should  include  help  (similar  to  that  for  examples). 

5.1.6  Are  Your  Test  Items  Adequate? 

The  IQI  manual  also  gives  some  rules  to  observe  when  you  construct  or  check  the 
quality  of  test  items.  Be  sure  that: 

1.  The  item  is  clear  and  not  ambiguous. 
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item. 


2.  The  item  does  not  give  away  its  own  answer- -or  the  answer  to  any  other 


3.  The  item  is  well  constructed. 

The  word  "ambiguous,"  used  in  Rule  1,  comes  from  two  Latin  words  meaning 
"both  ways"  and  "lead."  In  other  words,  to  lead  both  ways  at  once,  or  aimlessly,  which  is 
the  worst  kind  of  leading.  That  is  what  a  test  item  will  do  if  you  are  not  careful.  It  can 
lead  the  student  on  a  wild  goose  chase.  Here  is  an  example  of  poor  directions:  "Which  of 
the  following  is  able  to  provide  close  air  support?"  That  is  ambiguous,  because  the 
student  is  not  told  whether  to  pick  one  answer  or  all  the  answers  that  apply.  It  would  be 
better  to  say,  "Select  the  one  weapon  system  best  able  to  provide  close  air  support." 

Rules  2  and  3  are  self-explanatory,  but  not  always  easy  to  achieve.  To  make 
sure  that  none  of  your  test  items  gives  away  its  answer,  let  someone  else  look  at  your 
test.  Someone  else  will  see  your  mistakes  better  than  you  can.  Tell  your  "critic"  to  look 
especially  for  items  where  the  answers  can  be  figured  out  from  the  test  itself. 

5.1.7  Are  the  Parts  of  Your  Instruction  Consistent? 

So  far,  we  have  dealt  with  the  adequacy  of  objectives,  presentations,  examples, 
practice,  feedback,  and  tests.  Another  part  of  quality  control  is  consistency.  Read  the 
IQI  manual  to  learn  how  to  make  the  following  "big  three"  consistent  with  each  other: 

1.  Objectives. 

2.  Tests. 

3.  Presentations. 

In  other  words,  test  items  should  be  consistent  with  objectives  and  presentations  should  be 
consistent  with  both  objectives  and  test  items. 

Consistency  means  two  things.  For  example,  a  test  item  is  consistent  with  an 
objective  if  (1)  the  learning  task  performed  for  the  test  item  is  the  same  as  the  task 
stated  in  the  objective  and  (2)  the  type  of  content  the  student  must  learn  for  the  test  item 
is  the  same  as  the  type  of  content  in  the  objective. 

The  IQI  manual  shows  how  to  determine  the  learning  task  and  the  type  of 
content.  One  type  of  task  is  "remember"  and  one  type  of  content  is  "fact."  When  you  put 
these  two  together,  you  obtain  "remember  fact."  This  means  that,  if  the  objective  is  to 
remember  a  fact,  then  the  test  item  should  require  the  student  to  remember  a  fact  and 
the  presentation  should  teach  how  to  remember  the  fact. 

Figure  5-1  shows  the  entire  IQI  classification  scheme,  which  has  five  types  of 
content  and  three  tasks.  Notice  that  you  can  perform  only  one  task  (remember)  with  bare 
facts.  You  cannot  use  a  fact  by  itself.  Facts  are  usable  only  when  they  are  Unked  with 
other  facts  in  a  category,  procedure,  rule,  or  principle. 

Why  is  it  so  important  to  be  consistent?  If  the  presentation  prepares  the  student 
to  do  one  thing  and  you  test  the  student  on  something  else,  you  will  not  get  good  results. 
Suppose  your  objective  is  for  the  student  to  use  a  procedure,  your  presentation  teaches 
the  student  to  remember  a  fact,  and  you  test  the  student  on  using  a  principle.  How  well 
do  you  think  the  student  will  perform? 
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FACT  CATEGORY  PROCEDURE  RULE  PRINCIPLE 


Figure  5-1.  IQ1  classification  table. 


5.1.8  Obtaining  the  1Q1 

If  you  do  not  already  have  the  four  volumes  of  the  1Q1,  contact  the  Defense 
Technical  Information  Center,  Cameron  Station,  Alexandria,  VA  22314  (Telephone:  284- 
7633  (Autovon)  or  (202)  274-7633  (Commercial).  AD  numbers  are  provided  in  the 
reference  list. 

5.2  Management 

Individualized  courses  need  more  management  control  than  do  lock-step  courses. 
On  a  lock-step  course,  management  is  pretty  simple  because  everybody  does  the  same 
thing  at  the  same  time.)  Some  of  the  questions  you  will  need  to  answer  to  manage 
individualization  are: 

1.  How  will  students  know  what  to  do  next? 

2.  How  will  their  progress  be  controlled? 

3.  How  many  retests  will  be  allowed? 

4.  How  much  time  is  allowed  to  complete  a  lesson? 

Other  questions— that  you  never  thought  of— will  come  up  in  the  first  few  weeks  of  your 
individualized  course.  To  deal  with  them  all,  do  these  two  things:  (1)  Write  down  a  rule 
for  each  decision  that  you  make  and  (2)  print  a  student  progress  card  that  shows  how  the 
rules  apply.  Examples  are  provided  below. 

1.  If  you  decide  that  the  passing  score  will  be  90  percent  for  lesson  tests,  you 
could  show  that  right  on  the  student  progress  card.  For  most  rules,  it  can  show  the 
instructor  what  will  happen  next.  Figure  5-2,  which  is  part  of  a  student  progress  card, 
shows  what  will  happen  after  a  student  takes  the  lesson  test  for  the  first  time.  Notice 
how  the  passing  test  score  (90-100%)  is  printed  right  on  the  card.  What  will  happen  if  the 
student  scores  less  than  90  percent  is  shown  on  the  card,  too:  (a)  The  instructor  reviews 
the  student's  test  and  (b)  assigns  exercises  #50  through  #100. 

With  a  card  like  this,  the  Instructor  knows  Instantly  what  all  students  should 
do  next.  This  is  important,  since  you  do  not  want  to  waste  the  instructor's  time  with 
small  on-the-spot  decisions.  You  want  the  Instructor  to  be  guiding,  motivating,  tutoring, 
and  giving  feedback  to  students. 
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Figure  5-2.  Showing  passing  score  on  student  progress  card. 

2.  Suppose  you  decided  that  a  student  should  be  able  to  complete  exercises  #50 
through  #100  in  half  a  day.  That  is,  if  the  student  takes  longer  than  that,  there  may  be  a 
problem.  You  cannot  expect  the  instructor  to  remember  when  the  half  day  is  up.  It  would 
never  work.  An  instructor  would  have  a  hard  time  memorizing  a  dozen  different  times  to 
check  on  students'  progress. 

A  better  way  is  to  leave  space  to  enter  the  start  and  estimated  completion 
times  right  on  the  student  progress  card  (see  Figure  5-3).  The  EST.  FINISH  time  says  1430 
instead  of  1330  to  leave  time  for  lunch,  of  course.  Now  all  the  instructor  has  to  do  is  look 
every  half  hour  or  so  at  the  student  control  cards  and  at  the  clock  to  see  whether  the 
students  are  on  time  or  behind  schedule.  Of  course,  this  is  just  an  example.  For  your  own  ^ 

course,  you  would  probably  do  it  a  little  differently. 


How  will  the  students  know  how  long  to  study  a  tape/slide  lesson?  Can  they 
spend  all  day  at  it?  What  should  they  do  when  they  finish?  How  will  the  students  know 
these  things?  Should  you  leave  it  up  to  the  instructor  to  put  out  the  word?  No,  you  do  not 
want  to  waste  the  instructor's  time  this  way.  If  the  instructor  spends  all  of  his  or  her 
time  giving  directions,  there  would  be  total  confusion  in  the  class.  That  leaves  the 
instructional  materials.  Put  directions  to  the  student  right  in  the  content  of  the  lesson. 
Figure  5-4  shows  some  examples  of  directions  that  you  could  put  in  a  tape/slide  lesson, 
and  Figure  5-5,  in  a  printed  lesson. 
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5.3 


Management  Records 


In  the  last  section,  we  discussed  using  a  student  progress  card.  You  need  a 
progress  card  or  something  like  it  to  know  where  your  students  are  and  what  they  should 
do  next.  In  this  section,  we  will  look  at  examples  of  student  progress  cards  and  other 
management  records. 

Figure  5-6  is  a  control  card  for  a  worksheet  system  for  one  lesson  or  module. 
Each  time  the  student  starts  another  module,  you  would  add  another  card. 


NAME _ _ MODULE 


Figure  5-6.  Student  progress  card  for  the  worksheet  system. 


The  general  purpose  student  progress  card  shown  in  Figure  5-7  could  be  used  for 
any  instruction  system  that  used  several  tests  with  instruction  between  retests.  It 
includes  a  "test  review"  to  cover  any  errors  the  student  made,  even  if  the  student  passes 
the  test. 
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-i  ?  SON 


Figure  5-7.  General  purpose  student  progress  card. 


Figure  5-8  shows  a  student  progress  card  for  a  lesson  that  includes  a  diagnostic 
test  before  instruction.  In  this  particular  system,  the  students  study  only  the  lesson(s) 
that  the  diagnostic  test  indicates  they  should.  (Like  the  previous  student  progress  cards, 
the  card  shown  in  Figure  5-8  is  for  only  one  module  at  a  time.)  Finally,  Figure  5-9  shows 
a  student  progress  card  that  can  be  used  for  several  lessons.  (You  may  recognize  the 
names  of  the  lessons  as  those  at  IMS.) 

Remember,  these  are  just  examples.  In  your  own  course,  you  would  do  things 
your  way.  A  sample  student  progress  card  is  provided  on  p.  C-l  for  you  to  use  with  or 
without  revisions  in  your  course.  The  important  thing  is  to  "cover"  all  the  times  when  the 
instructor  or  the  student  has  to  know  what  to  do.  Do  not  leave  it  to  the  instructor  to 
figure  out  on  the  spot. 

It  is  impossible  to  think  beforehand  of  all  the  questions  you  will  need  to  answer. 
3ust  to  get  you  started,  Figure  5-10  lists  some  of  the  questions  you  must  answer  to 
manage  your  course.  For  each  item,  we  have  shown  where  you  need  to  provide 
instruction.  The  individualization  system  checklist  is  provided  on  p.  C-2  for  you  to  use 
when  you  individualize  your  course. 
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PRETEST  'LES-  START  FINISH  POST-TEST  START  FINISH  POST-  INSTRUCTOR 


i  i  i  i  i  i  i  '  i  i  r 

NOTE:  M  “  Mastery  NM  ■  Nonmastery 


Figure  5-9.  Student  progress  card  for  several  lessons, 


Questions/Decisions 

Tell 

Instructor 

On  Control 
Card 

Tell 
Student 
in  Lesson 

1. 

How  will  student  know  how  to  get  started? 

X 

2. 

How  will  students  know  what  to  do  next? 

X 

3. 

How  will  student  know  lesson  objectives? 

X 

4. 

Can  students  skip  the  lesson  pretest? 

X 

X 

5. 

Can  students  "challenge"  the  lesson  test? 

X 

X 

6. 

How  will  the  student  know  if  he  or  she  is 
ready  for  the  lesson  test? 

X 

7. 

How  will  the  instructor  know  if  the  student 
is  ready  for  the  lesson  test? 

X 

8. 

How  long  can  a  student  spend  on  a  particular  test? 

X 

X 

9. 

Is  there  a  minimum  time  to  spend  studying 
between  test  and  retest? 

X 

X 

10. 

How  long  may  a  student  spend  on  a  particular  lesson? 

X 

X 

11. 

What  is  the  criterion  for  passing  a  lesson? 

X 

X 

12. 

How  will  the  instructor  know  where  the  student 
is  located  in  the  program? 

X 

13. 

What  will  the  instructor  do  if  a  student  is  too  slow? 

X 

14. 

What  will  the  instructor  do  if  a  student  has  "free  time?" 

X 

15. 

How  will  the  instructor  know  that  a  student  needs 
extra  help? 

X 

16. 

How  will  the  student  know  he  or  she  needs  extra  help? 

X 

17. 

How  will  the  student  get  extra  help? 

X 

18. 

How  long  can  a  student  take  to  complete  the  course? 

X 

X 

19. 

What  is  the  criterion  for  passing  the  course? 

X 

X 

Figure  5-10.  Course  management  decisions. 


You  will  need  other  course  records,  too;  for  example,  you  need  a  way  to  see  at 
once  where  all  your  students  are.  (So  you  can  tell  If  you  need  seats  for  more  students,  for 
instance.)  Figure  5-11  is  an  example  of  a  wall  chart  showing  the  status  of  a  whole  class. 
To  make  a  chart  like  this,  you  could  use  a  chalkboard,  magnetic  board,  or  just  some 
cardboard  on  a  wall.  Look  on  the  chart  and  notice  these  things: 

1.  Allen  is  behind  schedule.  He  should  have  finished  lesson  05  by  now  (his 
eighth  day). 
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2.  Smith  is  beh.nd  schedule,  too.  She  should  have  finished  lesson  #1  by  now 
(her  second  day). 

3.  All  others  are  on  schedule  or  ahead  of  schedule. 


STUDENT  PROGRESS 


STUDENT 

toy 

1  2  3  4  S  6  7  8  9  10  11  12  13  14 

IS  16 

Allen 

. ®  m®  •  ®  •  ®  • 

•0 

Black 

. □  ‘  ®  •  ®  ‘  ®  • 

•0 

Jones 

□  0  0  *  0  *  (D 

*  0 

Jackson 

*  ‘  ED®®  ‘  ®  ‘  0  •  ®  • 

•0 

Snlth 

QED®®  •  ®  •  •  ®  •  0  •  0  * 

0 

1.  Numbers  on  round  tags  represent  lesson  numbers. 

2.  Rectangular  tag  Indicates  student's  status  by  program 
day. 

3.  Position  of  round  tags  indicates  the  program  day  on 
which  the  student  is  scheduled  to  finish  each  lesson. 
When  a  lesson  is  finished,  numbered  round  tag  is 
removed. 


Figure  5-11.  Example  of  wall  chart  showing  status  of  entire  class. 


Remember,  these  are  only  examples  of  management  records.  You  do  not  have  to 
use  these  exact  forms  in  your  own  course,  but  can  design  your  own  progress  cards  and 
control  chart.  The  same  information  is  needed,  no  matter  what  you  teach:  (1)  where  the 
student  is  and  (2)  what  the  student  should  do  next. 

5.4  Scheduling 

One  problem  that  comes  up  in  individualized  instruction  is  how  to  schedule  group 
activities.  Can  you  present  a  guest  speaker  or  hold  a  group  discussion  within  individu¬ 
alized  instruction?  You  can  by  reserving  a  block  of  time.  For  instance,  you  could  hold 
open  the  time  from  1300  to  1500  every  Wednesday.  Students  would  know  in  advance  that 
they  are  meeting  as  a  group  at  that  time.  You  could  hold  open  as  much  time  as  necessary 
this  way.  In  one  Air  Force  program,  for  example,  the  class  day  is  almost  evenly  divided 
between  group  and  individualized  instruction.  The  way  you  set  up  your  schedule  is  up  to 
you. 
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5.5  Summary 


Quality  control  and  course  management  are  even  more  important  in  individu¬ 
alized  instruction  than  in  traditional  instruction.  In  a  lock-step  system,  the  platform 
instructor  can  "fine-tune"  the  course  as  he  or  she  goes  along.  This  is  more  difficult  in  an 
individualized  instruction  system. 

You  need  to  pay  close  attention  to  the  effectiveness  of  your  instruction  and 
prepare  carefully.  Take  the  time  to  ask  yourself  all  the  questions  we  have  asked  in  this 
chapter  about  how  the  system  will  run.  Once  it  starts  running,  it  will  be  a  little  late  to 
start  thinking  about  how  students  will  progress. 

Write  down  each  management  decision,  such  as  how  long  a  student  will  have  to 
complete  the  course.  Then  act  on  these  decisions  by  putting  student  directions  right  into 
the  instruction.  Put  rules  that  the  instructor  should  follow  right  into  the  student  progress 
card.  Then  the  instructor  will  not  have  too  many  decisions  to  make  at  one  time. 

After  your  individualized  course  has  been  in  operation  several  weeks,  you  will 
probably  want  to  make  some  changes.  You  may  go  through  several  "editions"  of  student 
progress  cards  and  schedules  before  you  get  what  you  need.  The  same  is  true  for 
scheduling. 

As  you  must  start  with  some  rules  to  keep  from  getting  lost,  set  up  rules  for 
guiding  students  through  the  course.  You  can  change  these  rules  later  if  needed. 

6.0  THE  ROLE  OF  EVALUATION 

There  are  actually  many  reasons  to  evaluate  individuals.  You  are  probably 
reading  this  text  to  learn  about  evaluating  students  in  a  classroom.  The  most  common 
reason  is  probably  to  see  if  a  student  has  gained  the  skills  and  knowledge  you  are  teaching; 
in  other  words,  to  see  if  the  objectives  of  the  training  have  been  met.  Another  reason  is 
remediation --to  see  which  skills  and  knowledge  the  student  still  needs  to  acquire.  Third, 
you  might  evaluate  persons  to  qualify  them  for  promotion,  special  assignments,  profi¬ 
ciency  pay,  etc.  Fourth,  individuals  might  also  be  evaluated  to  estimate  the  combat 
readiness  of  a  whole  unit.  For  example,  the  proportion  of  Marines  who  can  perform 
certain  tasks  is  one  indicator  of  the  readiness  of  a  Marine  unit  as  a  whole  and  the 
percentage  of  vehicles  that  are  ready  to  roll  is  another. 

Although  this  instructional  package  is  aimed  at  evaluating  students  in  training, 
the  same  principles  apply  to  all  evaluations.  Therefore,  if  you  are  going  to  build 
performance  tests  for  the  field,  the  following  chapters  on  evaluation  should  also  be 
helpful  for  you. 

1.  Evaluation  and  individualization.  Individualization,  as  you  have  seen,  calls 
for  lots  of  testing.  You  need  to  know  how  each  student  is  doing  to  fit  your  instruction  to 
that  student,  especially  when  you  are  trying  to  get  each  student  to  the  same  level  of 
mastery.  If  you  do  not  evaluate,  you  do  not  know  if  the  student  has  reached  mastery. 

In  your  individualized  system,  the  purpose  of  evaluation  Is  to  affect  the 
student's  performance,  not  merely  to  find  who  is  the  best  in  the  class.  To  affect  the 
student  performance,  evaluation  must  meet  these  standards; 

a.  It  must  tell  the  instructor  how  the  student  is  doing. 
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b.  It  must  tell  the  student  how  he  or  she  is  doing. 

c.  Its  results  must  be  available  fast. 

As  the  results  of  evaluation  control  what  the  student  does  next,  they  must  be  available 
now,  not  tomorrow.  Also,  both  the  instructor  and  the  student  must  know  what  the  test 
results  mean. 


Compare  the  role  of  testing  in  a  typical  high  school  course  to  the  role  of 
testing  in  the  individualized  IMS  course.  In  a  high  school  course,  if  you  take  a  test  on 
Tuesday,  you  get  your  grade  on  Wednesday  or  Thursday.  In  the  meantime,  the  course  rolls 
on  to  the  next  topic.  At  IMS,  however,  you  get  your  test  results  much  sooner  and  you  do 
not  go  on  to  the  next  topic  until  you  pass  the  test. 

2.  Decisions  in  testing.  Usually,  a  test  cannot  cover  all  the  elements  of 
training  or  job  performance.  The  number  of  things  you  might  measure  is  usually  many 
times  the  number  of  things  you  can  measure.  Therefore,  you  have  to  decide  such 
questions  as: 

a.  What  should  be  tested? 

b.  How  much  should  be  tested? 

c.  What  kind  of  test  should  be  used? 

d.  Does  the  test  measure  what  it  should? 

Making  decisions  about  such  questions  is  what  evaluation  is  all  about. 

3.  Test  development.  The  development  of  a  test  consists  of  three  major 
activities:  (a)  planning  the  test,  (b)  constructing  the  test,  and  (c)  tryout/revision. 
Planning  the  test,  the  most  critical  step  of  the  three,  will  be  discussed  in  Chapter  7. 
Chapter  8  deals  with  test  construction,  and  Chapter  9  provides  guidance  on  trying  your 
test  out  and  revising  it. 

7.0  PLANNING  A  TEST 

Planning  a  test  involves  three  major  decisions.  The  first  is  what  to  test.  Before 
you  begin,  you  have  to  get  a  very  clear  picture  in  your  mind  of  what  you  are  trying  to 
measure  (this  is  not  always  easy).  Second,  you  have  to  decide  how  to  sample.  Sometimes 
you  cannot  test  everything  you  want  to  and  you  have  to  choose  certain  areas.  Third,  you 
have  to  decide  the  form  of  test  to  use.  This  choice  will  depend  a  lot  on  what  you  are 
measuring.  You  might  use  a  different  form  of  test  to  learn  how  well  a  student  performs  a 
procedure  than  to  test  a  problem-solving  skill. 

7.1  What  to  Cover  (The  Domain) 

Whatever  your  reason  for  evaluating  individuals,  you  first  have  to  define  what  is 
to  be  evaluated.  The  most  important  requirement  for  good  evaluation  is  a  clear,  precise 
definition  of  the  areas  of  achievement  to  be  measured.  For  example,  to  evaluate  students 
or  trainees  either  the  objectives  or  the  content  of  training  must  be  spelled  out.  To 
evaluate  job  performers,  the  acts  that  make  up  correct  job  performance  must  be  carefully 
stated  beforehand. 

The  specification  of  what  is  to  be  evaluated  is  called  the  performance  domain  by 
psychologists.  Ideally,  if  the  performance  domain  is  well  defined,  two  different  test 
developers— working  separately- -would  come  up  with  similar  tests.  That  would  also 
depend  on  other  things  such  as  agreeing  on  initial  test  specifications.  The  point  Is  you 
cannot  evaluate  something  unless  you  know  what  It  is.  (You  cannot  get  much  use  out  of 
test  results  either,  if  you  do  not  know  how  they  were  obtained.) 
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To  select  the  right  content  domain,  stop  and  think  about  the  purpose  of  your 
test.  To  measure  student  achievement,  the  training  objectives  will  be  your  content 
domain.  If  objectives  are  nonexistent  or  inadequate,  you  should  review  the  training 
content  to  define  clear  objectives.  (The  content  of  training  may  be  in  many  forms;  for 
example,  lesson  plans,  lecture  notes,  handouts,  audiovisual  materials,  texts.) 

To  measure  job  performance- -not  training  performance— job  performance  is 
your  domain.  Job  performance  would  have  to  be  spelled  out  in  terms  of  tasks  or  some 
other  statement  of  job  requirements. 

Sometimes  you  cannot  measure  what  you  want.  Maybe  it  costs  too  much  or 
measurement  is  unreliable.  A  good  example  is  something  like  "positive  attitude."  This 
trait  would  be  great  to  know  about  when  promoting  someone,  but  there  is  no  reliable  way 
to  measure  it  or  even  define  it.  Another  example  is  "total  job  proficiency."  Although  it 
would  be  great  to  know  an  individual's  total  job  proficiency,  it  is  impossible.  It  would  cost 
far  too  much  to  test  an  individual  on  every  single  thing  he  or  she  ever  does  on  the  job. 

When  you  cannot  measure  the  domain  that  you  want,  you  measure  what  you  think 
is  closest  to  the  domain  you  want.  In  the  case  of  "total  job  proficiency,"  you  can  measure 
part  of  an  individual's  job  proficiency  or  job  knowledge.  (Job  knowledge  is  not 
proficiency,  but  it  is  close  to  it.  You  cannot  do  something  if  you  do  not  know  how.) 
Knowledge  takes  less  time  to  measure.  When  the  domain  you  are  interested  in  is  too 
costly  to  measure,  you  have  to  consider  the  pros  and  cons  of  such  "intermediate"  measures 
as  sampling  the  domain. 

7.2  Sampling  Strategy 

Whatever  content  domain  you  define,  it  will  probably  have  more  elements  than 
you  can  measure.  Possible  units  of  test  performance  include  all  elements  that  you  could 
identify  in  a  close  look  at  the  content.  Since  training  and  job  domains  consist  of  so  many 
elements,  all  of  them  cannot  be  represented  on  tests. 

What  will  you  do?  The  answer  is  sampling.  Before  you  build  a  test,  you  decide 
how  to  sample  the  elements  in  your  content  domain.  Before  we  discuss  different  ways  to 
do  this,  let  us  consider  the  idea  behind  sampling. 

Consider  sampling  drinking  water  for  purity.  When  an  inspector  takes  a  sample 
of  water  for  testing,  is  the  inspector  interested  in  the  sample  in  hand?  No.  When  a 
doctor  takes  a  blood  test,  is  the  doctor  interested  in  the  blood  taken  from  your  arm?  No, 
of  course  not.  The  water  taken  for  sampling  will  never  be  drunk  and  the  blood  takfen  from 
your  arm  is  never  going  back  into  you.  The  inspector,  or  the  doctor,  is  interested  in  what 
is  left— all  the  water  in  the  total  water  supply  or  all  the  blood  in  your  body.  When  you 
take  a  sample  of  something,  you  are  interested  in  the  larger  thing  you  took  the  sample 
from,  not  in  the  sample  itself. 

If  a  sample  of  water  from  a  water  supply  is  pure  enough  to  drink,  we  conclude 
that  the  whole  water  supply  is  pure  enough.  If  a  sample  of  blood  from  your  body  is  type  O 
positive,  we  conclude  that  all  your  blood  is  type  O  positive,  not  just  the  blood  in  the 
sample.  Once  you  think  about  this,  it  is  obvious.  Nobody  gets  confused  about  sampling 
water,  because  water  is  mixed  very  well.  The  water  in  the  top  of  a  water  tank  is  about 
like  the  water  at  the  bottom. 

People  do  get  confused  about  evaluating  performance.  They  forget  that  almost 
nothing  can  be  evaluated  completely.  They  forget  that  evaluation  involves  sampling. 
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When  you  build  a  test,  you  are  interested  in  performance  beyond  the  test— not 
just  about  what  is  on  the  test  Itself.  Consider  the  whole  domain  you  are  trying  to 
measure.  When  you  sample,  you  want  to  select  elements  for  your  test  that  enable  you  to 
generalize  from  performance  on  the  test  to  performance  in  the  whole  domain. 

What  elements  will  you  include  on  your  test?  What  elements  will  best  represent 
the  whole  domain?  How  will  you  pick  them?  Your  sampling  strategy  consists  of  the 
answers  to  these  questions.  There  are  many  possible  sampling  strategies  you  could  use.  In 
this  section,  we  will  discuss  these  five: 

1.  Random  sampling. 

2.  Generality  of  content. 

3.  Frequency. 

4.  Performance  characteristics. 

5.  Criticality. 

7.2.1  Random  Sampling 

Selecting  elements  from  the  domain  at  random  is  often  very  reasonable.  This 
strategy  makes  no  assumptions  other  than  that  each  element  has  an  equal  chance  of  being 
selected.  It  does  not  depend  on  knowing  the  characteristics  of  the  elements  in  the 
domain.  Its  main  disadvantage  is  that  it  lacks  "power."  It  does  not  take  advantage  of  any 
information  about  the  characteristics  or  relationships  among  elements  in  the  domain  and 
these  might  be  important  for  your  purpose.  Often  you  can  improve  on  random  sampling  by 
classifying  the  elements  of  the  domain  using  one  of  the  strategies  described  below. 

7.2.2  Generality  of  Content 

When  commonalities  among  domain  elements  can  be  identified,  they  can  form 
the  basis  for  sampling  in  terms  of  the  generality  of  content.  Suppose  you  know  that 
information  about  the  layout  of  electronic  components  is  necessary  both  for  operating  and 
repairing  a  piece  of  equipment.  Then,  that  information  has  greater  generality  than 
information  needed  for  repair  work  alone.  If  you  can  classify  the  kinds  of  behavior 
underlying  job  performance,  you  can  sample  on  the  basis  of  generality. 

Unfortunately,  as  no  complete  classification  of  the  demands  of  jobs  exists, 
sampling  based  on  generality  must  be  based  on  judgments  about  commonalities  and 
dependencies.  Where  such  relationships  do  not  exist  or  have  not  been  identified,  you  may 
want  to  use  other  characteristics  of  the  domain  elements  as  a  basis  for  sampling. 

7.2.3  Frequency 

Sampling  the  content  domain  on  the  basis  of  frequency  can  often  be  done  fairly 
easily.  How  often  different  tasks  are  performed  in  a  job  can  usually  be  determined  from 
records  of  equipment  malfunction,  recollections  of  job  incumbents,  occupational  surveys, 
etc.  The  only  trouble  with  using  frequency  as  a  basis  for  sampling  is  that  you  cannot 
always  tell  what  it  means.  For  example,  Is  frequently  or  Infrequently  used  content  more 
important?  Tasks  that  are  frequently  performed  are,  obviously,  Important  to  the  job. 
You  can  argue  that  frequent  tasks  should  be  given  greater  representation  in  a  job 
proficiency  test.  On  the  other  hand,  because  frequently  performed  tasks  are  practiced 
more,  there  is  more  chance  that  a  person  can  do  them.  Therefore,  It  could  be  argued  that 
infrequently  performed  tasks  should  be  given  priority.  In  this  situation,  the  choice  comes 
down  to  judgment,  which  involves  other  factors  such  as  the  criticality  of  the  task  to  the 
mission. 
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7.2.4  Performance  Characteristics 


Other  bases  for  sampling  the  content  domain  are  characteristics  like  difficulty 
of  performance,  difficulty  of  learning,  and  variability  of  performance.  Although  this 
information  comes  from  records  of  success  and  failure  during  performance,  such  records 
are  often  not  available.  In  fact,  you  may  be  developing  your  test  to  get  such  information. 
When  you  do  not  have  data,  you  will  base  your  judgments  on  estimates  from  persons  on  the 
job.  In  those  cases,  the  standard  for  judging  difficulty  may  present  a  problem,  because  it 
may  be  hard  to  get  agreement  about  the  meaning  of  the  term  "difficulty." 

7.2.5  Criticality 

Another  basis  for  determining  the  most  important  elements  in  the  content 
domain  is  criticality.  Criticality  for  what  purpose?  For  example,  job  content  might  be 
evaluated  in  terms  of  its  contribution  to  (1)  task  completion,  (2)  the  neutralization  of 
different  types  of  targets,  (3)  mission  accomplishment,  or  (4)  battlefield  survival. 

7.3  Executing  Your  Sampling  Plan 

When  you  have  picked  one  basis  or  more  for  sampling  from  the  content  domain, 
sampling  can  begin.  No  single  approach  will  fit  all  situations.  If  you  have  selected  a 
single  basis  for  sampling,  like  frequency  of  use,  you  may  only  have  to  list  all  the  elements 
of  the  domain  on  the  basis  you  have  picked. 

Example:  If  ten  tasks  from  a  job  performance  domain  are  to  be  selected,  you 
might  choose  the  ten  most  frequently  performed  tasks  or  decide  to  include  some  less 
frequently  performed  tasks  of  lower  frequency.  If  so,  one  approach  often  used  is  to 
classify  the  tasks  into  several  lists.  You  could  select  a  group  of  frequently  performed 
tasks  and  a  group  of  tasks  that  fall  in  the  middle  of  the  range.  Then,  you  could  weight  the 
representation  of  tasks  from  the  different  classification,  such  as  six  high-frequency  tasks, 
three  middle-frequency  tasks,  and  one  low-frequency  task. 

Perhaps  you  have  defined  two  or  more  sampling  variables,  like  frequency  and 
criticality.  In  that  case,  it  is  often  convenient  to  set  up  a  sampling  table  or  matrix. 
First,  you  classify  the  tasks.  Then,  you  display  them  in  a  table,  perhaps  in  three  groups, 
as  in  the  last  example.  The  ten  from  the  high-frequency  group  with  the  highest 
criticalities  can  then  be  selected. 

7.4  Form  of  the  Test 


The  form  your  test  will  take  depends  a  lot  on  what  you  are  trying  to  measure. 
You  have  to  consider  the  content  of  the  performance  domain.  As  you  remember,  the 
performance  domain  might  be  training  objectives,  training  content,  job  performance 
requirements,  or  some  other  well-defined  content.  It  is  very  important  that  the  content 
of  the  performance  domain  be  clearly  defined.  Some  of  the  possible  forms  your  test 
might  take  are: 

1.  Performance  of  a  complete  unit  of  work. 

2.  Display  of  particular  sklll(s)  required  for  unit(s)  of  work. 

3.  Display  of  particular  knowledge  required  for  unit(s)  of  work. 

By  performance  of  a  complete  unit  of  work,  we  mean  a  task  or  job  activity  more 
or  less  as  it  would  occur  in  the  field.  Examples  of  a  particular  skill  are  deciphering  code 
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and  reading  a  compass.  Displays  of  knowledge  might  include  outlining  the  organization  of 
division  artillery,  designating  a  good  amphibious  task  force  objective,  or  developing  an 
electronic  warfare  plan. 

To  decide  which  test  form  or  combination  of  forms  is  best  for  your  purpose,  sort 
the  content  of  your  performance  domain  in  terms  of  four  questions: 

1.  Does  the  domain  contain  motor  skill  and/or  perceptual  skill?  Motor/per¬ 
ceptual  skill  is  sometimes  called  just  "skill."  We  are  ail  familiar  with  this  kind  of  skill: 
Ride  a  bicycle.  Shoot  a  bull's-eye  with  a  pistol.  Guide  a  TOW  missile.  Steer  a  truck 
cross-country.  Weld  a  seam.  All  these  activities  require  physical  coordination,  usually 
hand-to-eye  coordination. 

2.  Does  the  domain  contain  direct  application  of  procedures?  A  simple 
example  of  a  procedure  is  disassemblying  a  weapon,  which  is  usually  a  1-2-3  series  of 
steps.  Not  all  procedures,  however,  involve  hardware.  Cashing  a  check  and  preparing  a 
property  receipt  are  procedures.  When  we  say  direct  application  of  a  procedure,  we  mean 
following  each  step  exactly  as  spelled  out. 

3.  Does  the  domain  contain  application  of  rules  or  principles?  When  you  apply 
a  rule  or  principle,  you  are  using  general  guidance  to  do  something  specific.  You  are  not 
just  following  a  series  of  steps  that  have  been  laid  out  for  you.  A  good  example  is  target 
selection.  There  are  many  general  rules  about  what  kinds  of  targets  have  priority  that 
you  have  to  apply.  Two  officers  might  even  disagree  about  which  targets  to  attack  first, 
because  most  rules  allow  some  variation  in  application. 

4.  Does  the  domain  contain  problem  solving,  troubleshooting,  or  decision 
making?  In  these  kinds  of  performances,  you  often  have  to  figure  out  the  rules  yourself 
because  there  is  not  necessarily  a  general  rule  to  guide  you.  To  reach  a  decision,  you 
often  have  to  figure  the  pros  and  cons  of  the  choice  facing  you. 

To  decide  what  form(s)  of  test  to  use,  examine  your  content  domain  for  each  of 
these  four  kinds  of  performance.  Of  course,  you  might  find  out  that  you  have  all  four.  If 
your  domain  is  training  objectives,  for  example,  you  may  find  objectives  of  all  four  types. 

Naturally,  you  have  to  consider  practical  restrictions,  too.  Administering  hands- 
on  performance  tests,  for  example,  takes  a  lot  of  time  and  personnel.  Sometimes  you  do 
not  have  the  equipment  needed  for  the  best  form  of  test.  However,  considering  the  kinds 
of  performance  in  the  content  domain  is  the  place  to  start.  That  tells  you,  at  least,  what 
test  formats  can  be  used.  Let  us  see  how  these  four  kinds  of  performance  are  related  to 
different  forms  of  tests. 

7.4.1  Motor  and/or  Perceptual  Skill 

A  good  way  to  tell  whether  behavior  is  skilled  or  nonskilled  is  to  judge  whether 
or  not  practice  is  necessary  before  the  act  can  be  performed.  This  includes  acts  that  (1) 
are  hard  to  describe,  like  a  golf  swing,  (2)  are  learned  In  stages,  like  Morse  code,  and  (3) 
involve  timing  or  coordination,  like  tracking  a  moving  target.  Performance  that  cannot 
be  produced  just  by  telling  or  showing  someone  what  to  do  can  be  considered  skilled. 

If  the  training  or  job  performance  to  be  tested  Involves  motor  or  perceptual  skill, 
then  a  hands-on  (performance)  test  is  needed.  You  could  test  complete  units  of  activity 
like  placing  an  underwater  demolition  charge,  elements  of  required  skill  like  swimming  or 
welding,  or  elements  of  required  knowledge  like  reading  a  multimeter. 
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Direct  Application  of  Procedures 


The  type  of  training  or  job  performance  that  is  most  suitable  for  all  forms  of 
testing  is  the  direct  application  of  a  procedure,  such  as  starting  a  generator,  assembling 
equipment,  adjusting  or  operating  equipment,  and  so  forth.  Such  performance  can  be 
tested  through  (1)  a  paper-and-pencil  test  of  information  about  performance,  or  (2)  a 
hands-on-test  of  performance. 

1.  About  paper-and-pencil  tests.  Paper-and-pencil  knowledge  tests  are  more 
efficient  than  hands-on  tests,  relatively  inexpensive  to  construct  and  administer,  and  can 
pinpoint  particular  items  of  information  without  requiring  the  trainee  to  act  out  a  long 
string  of  steps.  Tests  of  knowledge  can  test  the  maximum  amount  of  crucial  information. 

Knowledge  tests  have  disadvantages,  too.  They  may  include  reading 
demands  that  are  not  present  in  the  job  or  test  irrelevant  information.  You  must  be 
careful  not  to  include  theoretical,  terminological,  and  general  information  the  job  itself 
does  not  require. 

Also,  paper-and-pencil  tests  often  fail  to  call  for  information  about  cues  and 
responses  that  are  found  in  job  performance.  Sometimes  it  is  not  even  possible  or 
practical  to  represent  all  the  important  cues  with  a  paper-and-pencil  test.  For  example, 
how  would  we  ask  whether  paint  is  thick  enough  to  apply?  This  is  hard  to  describe  in  a 
written  test.  Changes  in  response  requirements  are  even  more  likely.  Paper-and-pencil 
tests  almost  always  ask  the  student  to  select  the  correct  response,  but  the  task  on  the  job 
is  often  to  produce  the  correct  response. 

Paper-and-pencil  tests  (unlike  hands-on  tests)  cannot  represent  all  the  infor¬ 
mation  involved  in  performing  a  task.  (Example:  Open  hatch  by  grasping  handle,  turning 
handle  clockwise,  and  pulling  outward.)  Constructing  a  paper-and-pencil  test  always 
means  deciding  what  to  represent  and  what  to  leave  out.  Often  you  have  to  add  special 
cues  not  available  on  the  job  to  tell  the  students  what  information  you  want.  In  a 
multiple-choice  test,  for  example,  if  you  ask  when  to  do  something,  you  may  give  away 
information  about  other  aspects  of  the  task  as  well. 

Finally,  there  is  the  problem  that  the  behavior  required  by  paper-and-pencil 
tests  may  not  be  the  same  behavior  ordinarily  required  on  the  job.  For  instance,  it  has 
been  demonstrated  that  mechanics  perform  a  greater  number  of  equipment  checks  in 
paper-and-pencil  simulations  of  troubleshooting  than  they  do  on  the  job.  Presumably  they 
make  more  checks  on  the  paper-and-pencil  simulation  because  they  do  not  really  have  to 
set  and  calibrate  test  equipment  in  the  simulation. 

2.  About  hands-on  tests.  The  major  advantage  of  using  hands-on  tests  is  that 
they  tend  to  include  all  of  the  information  and  acts  required  in  actual  performance.  This 
happens  without  your  having  to  identify  each  element  of  performance  and  putting  it  in  the 
test.  On  the  other  hand,  hands-on  tests  have  drawbacks.  They  are  costly  and  time- 
consuming  to  construct  and  administer.  Because  you  usually  have  to  give  them  one-on- 
one,  the  number  of  tasks  you  can  evaluate  in  any  particular  situation  is  limited.  Someone 
has  to  observe  and  score  the  person  being  tested  or  the  product  of  the  test.  This  makes 
scoring  less  reliable,  because  different  scorers  tend  to  score  differently,  unless  they  are 
well  trained.  Well  constructed  checklists  reduce,  but  do  not  eliminate,  this  problem. 
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7.4.3  Application  of  Rules  and  Principles 


Testing  the  application  of  rules  and  principles  differs  in  at  least  one  big  way 
from  testing  the  direct  application  of  a  procedure:  As  rules  can  be  applied  in  many 
situations  and  principles  explain  a  variety  of  events,  their  applicability  to  a  situation  must 
first  be  recognized.  (Procedures  are  aimed  at  particular  situations,  and  their  applicability 
does  not  need  to  be  recognized.) 

Suppose  someone  has  to  put  out  a  fire.  He  or  she  needs  to  know  that  the  correct 
procedure  depends  on  the  type  of  fire  as  well  as  how  to  put  it  out.  This  is  a  case  of  having 
to  apply  a  rule.  The  rule  might  be  "On  paper  and  wood  fires,  use  water;  on  oil  fires,  use 
carbon  dioxide." 

Contrast  this  with  the  basic  procedure  for  assembling  an  M16  rifle,  which  is  a 
procedure  that  is  done  the  same  way  each  time.  You  do  not  have  to  learn  a  lot  of  rules  to 
apply  to  different  M16s.  You  follow  the  same  procedure  for  one  M16  as  you  would  for 
another  Ml 6. 

So  the  use  of  rules  and  principles  generally  requires  (1)  recognition  that  a  rule  is 
applicable  in  a  particular  situation  and  (2)  knowledge  of  the  rule.  But  the  use  of  a 
procedure  is  generally  suggested  by  the  situation  and  only  requires  knowledge  of  the 
procedures. 

This  difference  is  important  and  affects  the  way  you  test  tasks  involving  rules 
and  principles.  When  the  ability  to  apply  a  rule  is  to  be  tested,  the  test  has  to  give  the 
trainee  a  chance  to  reveal  awareness  of  the  relevance  of  the  rule  to  the  task.  Then,  the 
trainee  can  be  tested  for  knowledge  of  the  rule  itself. 

While  knowledge  of  a  rule  can  usually  be  tested  with  a  paper-and- pencil  test, 
testing  for  recognition  of  the  applicability  of  a  rule  is  more  difficult.  In  conventional 
multiple-choice  knowledge  tests,  just  asking  about  a  rule  is  enough  to  "cue"  it  as  a 
possibility.  Only  if  the  knowledge  test  (such  as  an  essay  test)  did  not  introduce 
information  that  would  not  be  available  on  the  job,  could  awareness  of  the  relevance  of 
rules  and  principles  be  tested.  When  this  is  not  possible,  the  best  way  to  ensure  valid 
testing  Is  to  use  a  hands-on  test  of  performance. 

7.4.4  Problem-solving,  Troubleshooting,  or  Decision-making  Tasks 

Tasks  that  are  often  labeled  as  problem-solving,  troubleshooting,  or  decision¬ 
making  present  special  problems  in  testing.  In  these  tasks,  the  facts  and  options  to  be 
taken  into  account  are  not  obvious.  They  must  be  thought  of  and  selected  by  the 
performer.  For  example,  the  person  attempting  to  solve  a  problem  has  to  think  of 
different  ways  that  might  lead  to  a  solution  and  the  troubleshooter  (unless  he  or  she  has  a 
troubleshooting  job  aid)  must  think  of  checks  that  will  eliminate  whole  areas  ("split-half' 
technique)  or  use  some  other  strategy,  such  as  checking  Items  with  high  probability  of 
failure.  Then  he  or  she  has  to  decide  which  components,  symptoms,  and  test  readings  to 
examine  to  carry  out  the  strategy.  The  decision  maker  may  have  to  think  up  likely 
possibilities  as  well  as  determine  a  basis  for  picking  one  of  them. 

The  freedom  of  performance  in  these  tasks  means  that  you  face  two  problems  In 
testing.  First,  a  variety  of  possible  features  of  the  test  can  easily  Influence  how  well  the 
student  performs.  Therefore,  when  you  are  testing  problem  solving  and  decision-making 
tasks,  you  have  to  avoid  any  differences  between  job  and  test  behavior  that  might  be 
caused  by  the  form  of  the  test  Itself. 
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Second,  the  test  of  problem-solving  and  decision-making  tasks  must  not  restrict 
or  define  performance  more  than  the  actual  job  situation.  This  is  because  an  important 
part  of  performance  in  these  tasks  is  being  able  to  pick  out  important  cues  that  are  not 
obvious.  The  best  approach  is  probably  to  try  to  reproduce  as  many  of  the  characteristics 
of  the  work  situation  as  possible  by  using  a  hands-on  performance  test.  It  is  probably  best 
to  test  problem-solving,  troubleshooting,  and  decision-making  performance  with  a  hands- 
on  test,  even  though  such  tests  may  also  include  elements  of  performance  that  are  not 
really  necessary  for  your  purpose. 

7.5  Summary 

To  plan  your  test,  you  have  to  make  three  decisions:  (1)  what  to  cover,  (2)  how 
to  sample  it,  and  (3)  what  form(s)  of  test  to  use.  The  decision  of  what  to  cover,  which  is 
called  defining  the  performance  or  content  domain,  is  the  most  important  step  in  planning 
your  test.  If  you  cannot  decide  what  you  want  to  test,  you  cannot  test  it  very  well. 

Since  you  cannot  cover  the  whole  performance  domain  in  a  single  test,  you  have 
to  sample  the  domain.  There  are  a  variety  of  strategies  you  can  use  for  sampling.  If  you 
consider  several  strategies,  you  can  be  more  deliberate  in  what  you  put  on  the  test.  If  you 
do  not  pick  a  sampling  strategy,  you  may  end  up  with  a  "hit-or-miss"  test. 

To  decide  what  form  of  test  to  use,  you  need  to  think  about  the  performance 
domain.  Does  it  contain  motor/perceptual  skill,  application  of  procedures,  rules,  or 
principles,  or  problem  solving?  The  answers  to  these  questions  will  help  you  decide 
whether  to  use  a  paper-and-pencil  knowledge  test  or  a  hands-on  performance  test. 

Remember  that  planning  your  test  is  the  most  important  part  of  test  develop¬ 
ment.  This  is  where  you  make  the  connection  between  the  test  and  the  performance  you 
want  to  measure.  If  that  connection  is  loose,  it  does  not  matter  how  well  you  construct 
the  test.  In  other  words,  it  does  not  help  to  measure  something  well,  if  you  are  measuring 
the  wrong  thing.  Be  ready  to  spend  a  lot  of  time  on  test  planning. 

8.0  CONSTRUCTING  A  TEST 

When  you  have  planned  what  kind  of  test  you  need,  you  can  start  to  build  it.  This 
chapter  covers  two  kinds  of  tests:  paper-and-pencil  knowledge  tests  and  hands-on 
performance  tests.  You  should  know  that  there  is  not  really  a  complete  distinction 
between  knowledge  and  performance  tests.  All  tests  measure  "knowledge,"  even  hands-on 
tests.  Paper-and-pencil  tests  often  measure  so-called  "performance." 

Example:  In  some  jobs  (e.g.,  instructor,  finance  officer,  computer  programmer, 
clerk,  or  intelligence  officer),  people  work  mostly  with  words  and  symbols.  For  jobs  like 
these  On  fact,  for  most  jobs),  paper-and-pencil  tests  can  represent  many  job  tasks.  If  the 
task  involves  paper-and-pencil  tasks  on  the  job,  then  a  paper-and-pencil  test  is  really  a 
performance  test. 

Also,  no  "performance"  test  is  really  a  test  of  performance.  The  person  taking 
the  test  knows  it  is  a  test,  not  a  work  situation.  There  Is  always  that  difference  between 
a  performance  test  and  real  performance.  Also,  most  performance  tests  omit  much  that 
would  be  Included  in  performance  on  the  job.  That  does  not  mean  the  test  is  bad,  but  that 
paper-and-pencil  and  hands-on  tests  are  both  representations  of  performance,  not 
performance  itself. 


I 


Paper-and- pencil  tests  and  hands-on  test  call  for  different  development  pro¬ 
cedures.  In  this  chapter,  we  will  describe  paper-and-pencil  test  construction  first. 

8.1  Constructing  Paper-and-pencil  Tests 

You  begin  the  construction  of  a  paper-and-pencil  test  by  writing  a  "pool"  of 
items.  The  item  pool  gives  you  alternate  items  to  diagnose  student  deficiencies,  a  basis 
for  making  alternate  forms  of  a  test,  and,  most  importantly,  extra  material  for  a  tryout. 
Then,  when  you  try  out  your  test,  you  can  throw  away  weak  or  defective  items. 

It  is  hard  to  tell  if  a  test  item  is  bad  just  by  looking  at  it.  It  takes  a  lot  of  skill 
to  see  what  meanings  students  will  "read  into"  an  item.  Therefore,  the  best  way  to 
identify  bad  items  is  to  make  up  a  trial  test  and  give  it  to  persons  with  the  same 
characteristics  as  the  ones  who  will  take  the  read  test  later.  This  way,  poorly  written 
items  show  up  fast. 

In  your  pool  of  test  items,  you  can  include  more  than  one  type  of  item.  The  pros 
and  cons  of  four  different  types  of  items  are  discussed  below.  These  are  not  the  only 
types,  though.  For  more  information,  review  your  IMS  lesson  on  developing  objective- 
referenced  tests  and  the  Handbook  for  Testing  in  Navy  Schools  (Ellis  &  Wulfeck,  1982). 

8.1.1  Free  Response  Items 

Free  response  items,  which  are  usually  cadled  "fill-in-the  blank,"  contain  blank 
space  for  the  answer.  The  answer  could  be  either  words  or  phrases.  This  is  the  only  type 
of  objective  test  item  in  which  the  student  must  produce  the  correct  response  instead  of 
merely  recognizing  it  on  a  list.  That  is  its  biggest  advantage. 

The  trouble  is,  free  response  items  are  hard  to  construct  and  score.  You  have  to 
ask  the  question  just  right  so  that  the  student  knows  exactly  what  answer  to  give.  This  is 
hard  to  do  without  giving  away  the  answer.  Even  when  you  ask  your  question  just  right, 
two  different  students  might  word  their  answers  differently,  which  causes  scoring 
problems.  When  different  words  are  used  by  different  students,  the  instructor  has  to 
judge  the  answers  and  objectivity  suffers. 

Follow  these  two  rules  when  you  write  free  response  items: 

1.  Do  not  take  statements  directly  from  a  text  and  use  them  as  free  response 
items.  Statements  removed  from  their  original  text  are  often  confusing. 

2.  Use  free  response  items  only  for  questions  that  can  be  answered  briefly  in  a 
few  words  or  a  few  sentences. 

8.1.2  True-false  Items 


Although  true-false  items  are  very  easy  to  write,  you  should  generally  avoid 
them  in  tests.  (It  Is  better  to  use  true-false  Items  only  for  study  questions  where  your 
goal  is  to  make  students  think.)  They  are  not  very  reliable  for  the  following  reasons.  If 
true-false  Items  are  just  "lifted"  from  a  manual,  the  sentence  Is  usually  either  printed 
word-for-word  ("true")  or  a  few  words  are  changed  to  make  It  "false."  This  only  tests  rote 
memory. 


If  you  use  true-false  items  to  test  knowledge  broader  than  facts,  you  run  Into 
another  problem.  Very  few  statements  are  all  true  or  all  false.  Test  constructors  often 
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"cover"  themselves  by  putting  in  qualifying  words  like  sometimes  and  never.  Well,  the 
student  can  often  spot  these  "qualifiers"  a  mile  away.  Good  true-false  items  that  do  not 
give  away  the  answers  are  very  difficult  to  write.  Follow  these  three  rules  when  you 
write  true-false  items: 

1.  Use  true-false  items  only  to  test  statements  that  are  always  true  or  false. 

2.  Do  not  use  true-false  items  to  test  statements  that  must  be  long  and 
complicated  to  be  universally  true  or  false. 

3.  Do  not  take  statements  directly  from  textual  material.  They  often  cease  to 
be  universally  true  or  false  when  removed  from  context. 

8.1.3  Matching  Test  Items 

In  matching  test  items,  the  student  is  presented  with  two  lists.  The  student 
picks  the  items  in  one  list  that  match  the  items  in  the  other.  Like  true-false  items, 
matching  items  are  good  for  testing  recall  of  names,  terms,  statements,  functions,  or 
other  facts. 

Watch  out  for  two  pitfalls  when  you  construct  matching  items.  First,  all  things 
on  your  list  should  be  of  the  same  type— all  people,  or  all  weapons,  or  all  terrain  features. 
Otherwise,  it  is  too  easy  for  the  student  to  figure  out  the  answers.  Second,  be  sure  any 
"wrong"  elements  are  reasonable  and  believable.  If  you  include  more  elements  in  one  list 
than  you  do  in  the  other  (a  good  idea),  be  sure  the  extra  elements  fit  in  with  the  real  ones. 

Follow  these  two  rules  when  you  write  matching  items: 

1.  Be  sure  that  the  basis  for  matching  is  clear. 

2.  Use  a  short  list  of  responses  to  match  to  the  list  of  cues.  Long  lists  always 
have  unimportant  elements  in  them.  The  purpose  of  the  item  is  to  test  knowledge,  not  the 
ability  to  keep  a  long  list  in  mind  while  making  a  choice. 

8.1.4  Multiple-choice  Items 

Multiple-choice  items  have  a  question  (or  "stem")  and  several  responses.  You 
can  ask  the  student  to  indicate  the  correct  response,  incorrect  response,  or  best  response. 
Multiple-choice  items  avoid  some  of  the  problems  of  other  types  of  objective  items.  They 
avoid  the  subjectivity  of  the  free-response  item,  the  absolute  truth  needed  by  the  true- 
false  item,  and  the  same-type  responses  needed  by  the  matching  Item. 

Follow  these  five  rules  when  you  write  multiple-choice  Items: 

1.  Experts  should  agree  on  the  correct  response.  Use  several  knowledgeable 
persons  to  review  the  item. 

2.  Avoid  negatively  stated  Items.  They  often  confuse  the  student. 

3.  Be  sure  your  incorrect  responses  (distractors)  are  believable.  Do  not  use 
distractors  that  are  obviously  wrong  even  to  someone  who  does  not  know  the  information 
being  tested. 
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4.  Do  not  use  responses  that  overlap  such  as:  'The  gas  tank  of  a  1/4  ton  truck 
holds  (a)  less  than  25  gallons  or  (b)  less  than  20  gallons.” 

5.  Locate  the  correct  answer  randomly  among  the  response  positions. 

8.1.5  Test  Item  Review 

Have  every  item  you  write  reviewed  by  another  subject  matter  expert.  There 
are  several  reasons  for  this  practice.  First,  a  person  who  writes  a  test  item  cannot  judge 
how  clear  it  will  be  to  others.  (It  always  seems  clear  to  the  item  writer!)  The  best  way  to 
see  if  the  item  is  clear  to  someone  else  is  to  give  it  to  another  person  for  review. 

Second,  you  need  someone  else  to  tell  you  if  the  item  is  really  pertinent  to  the 
job.  It  is  well  known  that  many  paper-and-pencil  test  items  are  not  relevant,  but  they  get 
into  tests  because  they  were  easy  to  write.  The  more  experts  who  look  at  your  test  items, 
the  better  the  chance  to  "weed  out"  items  that  are  not  relevant. 

Third,  you  need  a  "second  opinion"  about  this  question:  Is  the  paper-and-pencil 
format  really  adequate  for  this  task?  Earlier,  you  decided  that  the  paper-and-pencil 
format  was  appropriate.  Now  that  you  have  written  some  items,  double  check  your 
decision.  Is  the  item,  as  written,  a  good  indicator  of  whether  the  trainee  can  perform?  If 
you  think  so,  find  out  if  other  subject  matter  experts  agree. 

8.2  Constructing  Performance  Tests 

A  performance  test  is  sometimes  called  a  work-sample  test,  because  it  requires 
the  trainee  to  perform  a  sample  of  the  activity  from  the  work  situation.  Of  course,  a 
work  sample  rarely  consists  of  work  exactly  as  performed  in  a  job.  The  work  sample  is 
going  to  be  scored  and  the  trainee  knows  it.  The  conditions  and  responses  of  the  work 
situation  have  been  changed,  too.  But  the  sample  of  work  should  represent  important 
features  of  the  job.  In  addition,  a  performance  test  must  be  administered  in  a 
standardized  way. 

To  construct  a  performance  test,  you  need  to  answer  five  questions: 

1.  What  will  be  scored? 

2.  How  will  performance  be  scored? 

3.  How  will  the  test  be  set  up? 

4.  How  will  the  student  know  what  to  do? 

5.  How  will  the  scorer  know  what  to  do? 

We  will  address  these  questions  one  at  a  time. 

8.2.1  What  to  Score 

The  two  basic  types  of  performance  tests  are  (1)  process- scored  and  (2)  product- 
scored.  First,  you  need  to  figure  out  is  which  kind  of  task  you  have. 

"Perform  an  about-face"  is  an  example  of  a  process-scored  task.  It  should  be 
process-scored  because  the  scorer  must  watch  the  whole  movement  while  it  is  being  done. 
There  is  no  other  way  to  find  out  if  it  was  done  right.  Some  other  examples  of  actions 
that  should  be  process  scored  include  performing  the  manual  of  arms  and  playing  "Taps." 
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'Tie  a  square  knot"  can  be  product  scored  because  the  scorer  does  not  have  to 
see  the  knot  being  tied.  If  there's  a  square  knot  in  the  rope  at  the  end  of  the  test,  then 
the  student  has  tied  a  square  knot.  That  is  all  there  is  to  it. 

Lots  of  actions  can  be  scored  either  way;  for  example,  change  a  tire,  assemble  a 
rifle,  set  up  a  Claymore  mine,  etc.  That  is,  if  you  perform  all  the  steps  correctly,  then 
you  always  end  up  with  the  correct  product. 

Use  product-scoring,  if  you  can,  as  it  is  easier  to  administer  and  standardize. 
Sometimes  the  scorer  can  test  more  than  one  person  at  a  time.  Look  hard  at  the  task  or 
skill  you  want  to  test  to  see  if  it  can  be  product-scored.  (Many  tasks  that  can  really  be 
product-scored  look  as  though  they  have  to  be  process-scored,  because  they  have  usually 
been  described  as  a  process,  a  series  of  steps.)  Often,  for  example,  you  can  test  with  a 
function  check  (product-scoring).  You  do  not  have  to  see  the  weapon  being  assembled 
(except  sometimes  to  check  for  safety  rules).  Another  example  is  that  you  could  perform 
a  function  check  to  see  if  a  radio  was  set  up  correctly.  These  obvious  examples  illustrate 
how  you  should  look  at  your  tasks  to  see  if  you  can  use  product-scoring. 

8.2.2  How  to  Score  Performance 


Performance  will  be  scored  by  the  actual  performance  measures  that  make  up 
the  body  of  your  performance  test.  Whether  these  will  be  process- scored  or  product- 
scored  measures  will  depend  on  the  task  or  skill  you  are  testing.  Either  way,  you  need  to 
tell  the  scorer  what  to  look  for  and  the  standards  of  acceptable  performance.  Fip  ire  8-1 
is  an  example  of  process- scored  performance  measures  for  folding  the  U.S.  flag.  (Folding 
the  flag  could  also  be  product-scored.)  Figure  8-2  is  a  set  of  product-scored  measures  for 
baking  a  jelly  roll. 


FOLDING  THE  UNITED  STATES  FLAG 

GO  NO  GO 


A.  Folded  lower  striped  section  over 
blue  field  (1st  lengthwise  fold). 

B.  Folded  the  folded  edge  over  to  meet 
the  open  edge  (2nd  lengthwise  fold). 

C.  Started  triangular  fold  by  bringing 
striped  corner  of  the  folded  edge  to 
the  open  edge. 

D.  Repeated  folds  until  entire  length 
of  flag  was  folded  into  a  triangle 
with  only  the  blue  field  and  margin 
showing. 

E.  Tucked  margin  into  the  pocket  formed 
by  folds  at  blue  field  edge  of  flag. 


Figure  8-1.  Example  of  a  process-scored  performance  measure. 
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BAKED  3  ELLY  ROLL 


GO  NO  GO 


A. 

Role  is  cut  in  20  slices. 

B. 

Powdered  sugar  is  evenly  spread. 

C. 

Jelly  is  evenly  spread. 

_____ 

D. 

Roll  is  tightly  rolled. 

_____ 

E. 

Roll  is  baked  all  the  way  through. 

— 

Figure  8-2.  Example  of  a  product-scored  performance  measure. 


Performance  measures  come  from  well  stated  objectives  rigorously  derived  from 
task  analysis.  The  importance  of  good  task  analysis  and  clear,  complete  objectives  really 
shows  up  when  you  start  to  develop  tests.  The  elements  of  task  performance  must  be 
clearly  spelled  out  so  you  can  tell  whether  the  student  has  performed  correctly. 

As  the  rules  for  constructing  process- scored  measures  differ  a  little  from  those 
for  constructing  product- scored  measures,  two  sets  of  rules  are  presented  in  the  following 
pages.  In  both  types  of  measures,  you  tell  the  scorer  what  to  look  at  and  how  to  judge 
what  he  or  she  sees.  To  do  this,  you  sometimes  have  to  refine  the  statements  from  job  or 
task  analysis. 

8.2.2.1  Rules  for  Process-scored  Measures 


1.  Be  sure  the  standard  is  clear.  Before  you  use  the  action  "as  is"  from  the 
task  analysis,  double  check  to  be  sure  that  it  includes  the  standard.  Standards  answer 
questions  like  "how  much"  or  "how  well."  Be  suspicious  of  words  such  as  turn,  adjust, 
tighten,  and  others  that  call  for  judgment  and  words  ending  in  "ly"  such  as  properly, 
rapidly,  and  firmly.  They  might  be  scored  differently  by  different  scorers. 

2.  Put  only  one  action  in  each  scorable  measure.  Do  not  confuse  the  scorer; 
for  example,  "Removes  clip  and  pulls  slide  to  the  rear"  may  be  confusing.  What  about  the 
soldier  who  removes  the  dip,  but  does  not  pull  the  slide  to  the  rear?  You  are  asking  the 
scorer  to  judge  which  is  more  important.  Separate  any  combined  performance  measures 
you  find. 


Note.  Simultaneous  actions  are  OK.  "Depresses  accelerator  while  engaging 
starter"  is  acceptable  because  it  can  be  scored  as  right  or  wrong.  Stating  how  a 
step  is  performed,  such  as  "Aligns  sight  by  traversing,  elevating,  or  depressing 
gun  tube,"  is  also  acceptable. 

3.  List  only  necessary  actions.  Do  not  state  an  action  if  it  depends  on  other 
actions;  for  example,  "Releases  safety  cover;  activates  firing  switch."  If  the  firing  switch 
cannot  be  activated  without  releasing  the  cover,  you  can  omit  the  first  element.  Be 
careful  not  to  overwhelm  the  scorer  with  performance  measures.  The  scorer  cannot 
observe  the  student  while  marking  the  score  sheet. 


64 


it.  Be  sure  the  action  is  observable.  An  eye  or  head  movement  might  not  be 
observable  as  a  physical  act.  For  example,  'Throttle  is  in  idle  position"  can  be  evaluated 
only  if  the  throttle  is  placed  in  an  incorrect  position  and  the  student  has  to  move  it. 
Similarly,  "Checks  radar  scope"  may  actually  mean  the  student  has  to  turn  and  look  at  the 
scope.  In  a  case  like  this,  you  should  cue  the  scorer  to  the  performance  by  stating,  'Turns 
and  looks  at  scope  //I  before  activating  scanner." 

5.  State  error  limits.  Many  standards,  especially  with  measurements,  allow  a 
range  of  accuracy;  for  example,  "700-800  rpm"  or  "within  50  meters"  tell  the  scorer  the 
range  allowed. 

6.  State  safety  measures.  If  safety  is  a  factor,  write  a  safety  performance 
measure  stating  what  actions  must  or  must  not  be  done.  The  scorer  should  intervene  to 
prevent  the  violation  if  possible.  Be  specific.  Statements  like  "Commits  no  safety 
violations"  are  not  acceptable. 

Example  of  acceptable  safety  measures 


NO 
GO  GO 

a.  Examinee  removes  watch  and  rings,  if  any, 
before  working  on  radar  chassis. 


7.  State  the  time  limit.  If  the  task  has  a  time  limit  (e.g.,  "Mask  in  9  seconds"), 
it  is  a  separate  performance  measure.  Make  sure  both  the  scorer  and  the  student  know 
when  to  start  and  stop.  If  the  task  has  no  time  limit,  you  have  to  state  one  for  efficient 
test  administration.  In  this  case,  the  time  limit  is  not  meant  to  be  the  deciding  factor  in 
whether  a  student  is  "GO"  or  "NO  GO."  Talk  to  job  experts  and  conduct  field  tryouts  to 
arrive  at  a  reasonable  time  limit. 

8.  State  the  sequence.  Look  at  the  sequence  of  the  task  elements  carefully. 
Some  tasks  require  all  steps  to  be  in  order;  and  others,  only  some  steps.  If  a  particular 
order  is  required,  state  it  as  a  separate  performance  measure.  Tell  the  scorer  on  the 
score  sheet  if  order  is  not  important. 

S.2.2.2  Rules  for  Product-scored  Measures 


1.  Define  an  acceptable  product.  This  is  like  the  standard  of  performance. 
The  standard  in  typing  correspondence  may  be  that  a  letter  is  in  the  right  format  and 
contains  no  strikeovers  or  misspelled  words.  An  acceptable  product  may  be  a  condition 
such  as  an  engine  that  runs  or  a  telephone  that  rings.  It  may  be  an  action  such  as  a 
weapon  function  check  or  it  may  be  something  produced  such  as  a  completed  form  or  a 
computer  program.  When  you  are  developing  the  test,  carefully  examine  products  for  all 
dimensions  and  define  all  standards  required  for  scoring.  Examples  of  various  dimensions 
Include  format,  tightness,  tolerance,  shape,  location,  position,  time,  size,  and  adjustment 
limits. 


2.  State  error  limits.  See  process-scored  measures,  rule  5. 
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3.  State  the  time  limit.  See  process-scored  measures,  rule  7. 

4.  State  whether  scoring  aids  can  be  used.  Scoring  aids  are  used  to  help 
standardize  what  is  being  scored  such  as  overlays,  templates,  diagrams,  photographs  of 
correct  product,  and  photographs  showing  unacceptable  characteristics.  Scoring  aids  do 
take  time  to  prepare,  but  consider  using  them  when  words  alone  do  not  convey  an 
accurate  understanding  of  the  standard.  Example:  The  task  "Prepare  route  overlay"  leads 
to  a  product  that  is  difficult  to  describe  in  words,  but  can  be  clearly  shown  by  example. 
(You  may  also  find  scoring  aids  useful  in  scoring  process-scored  measures.) 

8.2.3  Test  Setup 

The  test  setup  is  stated  by  the  test  conditions,  which  include  environmental, 
equipment,  and  test  station  conditions.  First,  determine  what  the  conditions  should  be 
and  then  translate  them  into  instructions  to  the  scorers.  Your  goal  is  to  give  the  test 
exactly  the  same  way  to  every  student  every  time. 

1.  Environmental  conditions.  Environmental  conditions  apply  mostly  to  tasks 
performed  outdoors.  You  will  mostly  be  concerned  about  adverse  conditions.  If  one 
student  is  tested  in  the  rain  and  another  is  tested  in  the  sunshine,  will  the  test  be  fair?  If 
it  is  not,  specify  acceptable  conditions  of  visibility,  temperature,  precipitation,  light 
level,  etc. 


2.  Equipment  conditions.  Equipment  conditions  are  among  the  most  important 
aspects  of  hands-on  testing.  Again,  your  concern  is  to  standardize  the  test  conditions  for 
all  students.  List  all  items  of  equipment  needed  for  the  test,  including  publications. 
Then,  consider  all  aspects  of  the  equipment  itself  such  as  positions  of  switches,  dials, 
controls,  and  valves.  What  faults  should  be  built  in?  How  should  the  equipment  be  set  up 
for  each  student? 

3.  Test  station  conditions.  Test  station  conditions  depend  on  the  task.  Some 
can  be  performed  anywhere,  while  others  cannot.  Boresighting  a  large-caliber  gun  may 
require  a  line-of-sight  of  over  1000  meters.  At  a  minimum,  consider  these  factors: 
indoor/outdoor,  shop/workbench/ field  range,  distance,  and  terrain  and  vegetation. 

8.2.<f  How  the  Students  Know  What  to  do 


You  will  write  a  set  of  examinee  instructions  to  tell  the  students  what  they  are 
to  do.  In  hands-on  testing,  the  scorer  usually  reads  the  instructions  to  the  students; 
therefore,  you  will  be  writing  a  "script"  for  the  scorer  to  follow  word-for-word. 

Be  concise  and  direct  when  you  write  the  instructions.  Your  instructions  must  be 
easily  understood  and  complete  so  that  the  scorer  will  not  have  to  explain  or  answer 
questions  before  the  test.  Students  hearing  them  should  get  the  picture  right  away  to 
prepare  mentally  for  the  task.  Cover  the  following  areas  in  your  instructions: 

1.  State  the  job  situation.  Examples:  "You  are  conducting  a  weapons 
inspection."  "You  are  the  S-3  of  a  tank  battalion."  "This  Marine  has  a  broken  arm."  Do 
not,  however,  tell  the  students  what  they  are  supposed  to  figure  out  by  themselves.  To 
say,  "You're  setting  up  a  Claymore  mine  and  you  haven't  made  the  circuit  check  yet" 
would  give  away  the  fact  that  a  circuit  check  should  be  performed. 
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2.  State  what  the  students  are  to  do.  You  may  have  to  include  a  statement 
about  how  far  the  student  should  go.  (Example:  "Do  not  enter  the  control  number  in  the 
log.")  Be  careful  not  to  tell  the  student  how  to  do  the  task.  Do  not  say,  "Put  on  your 
protective  mask  and  clear  it."  The  student  should  remember  to  clear  it  himself. 

3.  State  the  time  and  accuracy  limits.  Example:  "You  must  complete  the  task 
within  20  minutes."  "Your  final  charge  must  be  plus  or  minus  1/8  pound."  Be  careful  not 
to  "cue"  the  student  as  to  what  to  do. 

4.  State  what  assistance  will  be  provided.  Tell  the  student  if  assistance  wiil 
not  be  provided.  If  there  will  be  help,  say  what  kind.  Examples:  "I  will  issue  the  fire 
command  to  you."  "I  will  position  the  aiming  stakes  as  you  direct  me." 

8.2.5  How  the  Scorer  Knows  What  to  do 

To  give  each  student  a  fair,  standardized  test,  you  need  to  write  clear 
instructions  to  the  scorer.  Be  precise  and  do  not  leave  things  up  to  the  scorer's  "best 
judgment."  Tell  the  scorer  exactly  what  to  do.  At  a  minimum,  include  the  following 
directions: 

1.  State  the  requirements  for  setting  up  the  test  station.  If  flammable 
materials  have  to  be  removed,  say  so.  If  the  area  has  to  be  free  of  visual  distractions  that 
normally  are  there,  say  so.  Tell  how  to  set  equipment  faults,  switches,  etc.  Tell  the 
scorer  just  how  to  set  up  the  test  station  using  a  CUagranvif  needed. 

2.  State  how  many  students  can  be  tested  at  once.  Examine  the  task  to 
determine  how  many  students  should  be  tested  together.  Usually  the  maximum  is  four, 
but  for  a  process-scored  test  you  can  generally  test  only  one  at  a  time. 

3.  State  what  kind  of  help  the  scorer  can  give,  if  any.  If  the  scorer  is  to  act  as 
a  team  member,  tell  how.  If  the  scorer  is  to  intervene  on  safety  violations,  tell  when  and 
how. 


4.  State  time  limits.  Tell  the  scorer  the  time  limits,  including  how  to  time  the 
test,  what  to  do  when  the  student  goes  over  the  time  limit,  and  perhaps  a  reminder  that 
time  limit  means  limit. 

8.3  Summary 


Paper-and-pencil  tests  and  performance  tests  call  for  different  methods  of 
construction.  You  start  developing  a  paper-and-pencil  test  by  writing  a  pool  of  items, 
because  you  will  discard  many  items  when  you  try  them  all  out.  Developing  a 
performance  test  starts  with  writing  performance  measures  based  on  objectives  derived 
from  task  analysis. 

This  chapter  presented  the  rules  for  constructing  both  kinds  of  tests,  including 
rules  for  different  types  of  test  Items  and  for  process-scored  and  product-scored 
performance  tests.  Use  multiple-choice  items  in  paper-and-pencil  tests  when  you  can.  In 
hands-on  tests,  use  product-scoring  if  possible.  These  are  general  rules.  In  some  cases, 
you  will  need  to  use  the  other  kinds  of  test  items  and  measures  discussed  here. 

Developing  good  tests  can  be  a  long,  hard  process.  The  rules  in  this  chapte-  are 
adequate  enough  to  guide  you,  but  there  is  no  substitute  for  lots  of  practice.  You  should 
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study  test  construction  because  individualization  calls  for  more  testing  than  does  lock- 
step  instruction.  Be  prepared  to  develop  a  lot  of  tests  if  you  are  going  to  individualize. 

9.0  TEST  TRYOUT  AND  IMPLEMENTATION 

After  planning  and  constructing  your  tests,  you  need  to  try  them  out.  After 
tryout  and  revision,  you  can  implement  them  in  your  course.  In  this  chapter,  you  will 
learn  how  to  try  out  paper-and-pencil  test  items  and  hands-on  performance  measures. 
Finally,  you  will  learn  about  implementing  your  tests  in  an  individualized  instruction 
system. 


9.1  Tryout  of  Paper-and-pencil  Tests 

The  goal  of  your  tryout  is  to  discover  poor  items  and  instructions.  Then,  you  can 
discard  or  revise  the  poor  items,  revise  the  instructions,  and  end  up  with  a  better  test.  A 
poor  test  item  (1)  has  no  correct  answer,  (2)  has  more  than  one  correct  answer,  (3)  is 
confusing  (ambiguous),  (4)  is  irrelevant,  (5)  does  not  agree  with  doctrine,  or  (6)  does  not 
separate  masters  from  nonmasters. 

Before  you  begin  any  tryout,  show  your  test  items  to  instructors,  test  developers, 
or  other  subject  matter  experts.  Ask  them  to  criticize  the  items  and  locate  factual 
errors.  Let  them  answer  the  items  to  see  if  your  directions  are  clear.  This  "pre-tryout" 
can  eliminate  most  of  the  obvious  errors  in  the  items. 

After  you  have  corrected  the  errors  your  "critics"  found,  you  are  ready  to 
conduct  the  formal  tryouts,  which  will  be  done  in  two  phases:  (1)  expert  tryout  and  (2) 
validation. 

9.1.1  Phase  1.  Expert  Tryout  of  Paper-and-pencil  Tests 

For  this  phase,  you  will  need  at  least  three  experts  in  the  area  you  are  testing. 
If  you  only  use  two  and  they  disagree,  you  would  not  know  who  is  right.  Using  three 
experts  does  not  indicate  whom  to  rely  on,  but  it  does  provide  a  way  to  settle 
disagreements  by  majority  opinion.  Do  whatever  you  can  to  get  high-level  performers  for 
the  expert  tryout.  You  do  not  want  recent  graduates,  no  matter  how  well  they  did  in 
school.  You  want  experienced  job  performers,  perhaps  senior  NCOs  or  officers.  Consider 
asking  some  experts  with  whom  you  usually  do  not  work,  as  they  may  be  more  critical. 

The  purpose  of  your  expert  tryout  is  to  find  all  test  items  that  are  poorly 
constructed  and  do  not  agree  with  doctrine.  (You  will  see  if  the  items  separate  masters 
from  nonmasters  in  Phase  2.)  First,  give  the  test  to  the  experts.  Then,  have  them  fill  out 
a  questionnaire  to  help  you  decide  what  to  do  about  each  item.  If  an  expert  fails  an  item, 
you  will  find  out  why.  Follow  these  steps. 

1.  Administer  your  test  to  the  experts.  (Do  not  tell  them  the  answers  until 
after  step  2.)  You  can  give  them  the  test  all  at  the  same  time  or  separately. 

2.  Give  the  experts  the  expert  tryout  questionnaire  shown  in  Figure  9-1.  (It  is 
also  provided  on  p.  C-3  for  you  to  reproduce  as  needed  for  your  course.) 


1 .  Do  you  think  you  answered  each  item  correctly? 

A.  Yes.  B.  No.  C.  Not  Sure. 

(Go  to  4.)  (Go  to  2.)  (Go  to  3.) 

2.  Which  items  do  you  think  you  did  not  answer  correctly? 

3.  Which  items  are  you  unsure  about? 

9.  Is  this  test  realistic?  (Does  it  reflect  how  the  job  is  done?) 

A.  Yes.  B.  No.  C.  Not  sure. 

5.  Were  the  words  or  phrases  in  the  test  easy  to  understand?  (If 
"No,"  indicate  in  the  space  below  the  words  or  phrases  that  gave  you 
trouble.) 

A.  Yes.  B.  No.  C.  Not  sure. 

6.  Are  all  of  the  alternatives  to  each  item  believable?  Of  "No," 
indicate  the  alternatives  that  are  not  believable.) 

A.  Yes.  B.  No.  C.  Not  sure. 

7.  Are  all  the  items  relevant  to  the  job?  (If  "No"  or  "Not  sure," 
indicate  the  items  that  should  be  deleted.) 

A.  Yes.  B.  No.  C.  Not  sure. 

8.  As  the  test  is  now  written,  do  you  think  it  is  a  fair  measure  of  a 
student's  ability  to  perform  on  the  job? 

A.  Yes.  B.  No.  C.  Not  sure. 

Figure  9-1.  Expert  tryout  questionnaire. 


3.  Go  over  the  test  items,  one  at  a  time.  The  first  sign  that  an  item  is  well 
constructed  and  agrees  with  doctrine  is  that  all  experts  get  it  right.  If  any  expert  fails  an 
item,  find  out  why.  Maybe  the  expert  was  not  as  expert  as  you  thought.  More  likely,  your 
test  item  is  at  fault.  It  might  be  misleading. 

Use  question  if  l  on  the  expert  tryout  questionnaire  (Figure  9-1)  as  your  main 
guide.  If  an  expert  thinks  he  failed  an  item  and  did  fail  it,  he  has  probably  forgotten  the 
procedure.  The  item  may  be  OK.  If  he  is  not  sure  of  his  performance  (answer  C)  and  then 
he  passed,  the  item's  wording  may  have  a  defect,  or  one  "incorrect"  response  may  be  as 
correct  as  the  correct  alternative.  When  an  expert  thinks  he  passed  an  item  that  he 
failed,  the  item  may  go  against  doctrine  or  the  "correct"  response  may  be  wrong  (or  at 
least  no  more  right  than  one  of  the  other  responses). 

The  second  sign  that  items  agree  with  doctrine  is  that  experts  think  the  test  is  a 
fair  measure  of  a  student's  ability  to  perform  on  the  job  (question  ffi,  answer  A).  Any 
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other  response  to  that  question  indicates  that  the  test  is  not  yet  acceptable.  Interview 
each  expert  who  responds  to  question  #8  with  answers  B  or  C.  After  the  expert  tryout, 
revise  all  the  items  that  had  deficiencies.  If  a  lot  of  changes  were  required,  you  should 
think  about  going  through  the  expert  tryout  again  with  three  new  experts. 

Suggested  actions,  which  are  keyed  to  the  questions  and  answers  on  the  expert 
tryout  questionnaire,  are  listed  on  p.  C-4. 

9.1.2  Phase  2.  Validation  of  Paper-and-pencil  Tests 

You  will  find  out  if  your  test  separates  masters  from  nonmasters  during 
validation.  "Validation"  in  the  world  of  testing  means  seeing  if  performance  on  the  test  is 
a  good  indicator  of  anything  else.  Why  is  validation  important?  After  all,  we  just  revised 
the  test  after  three  experts  took  it.  What  more  is  necessary? 

Well,  so  far,  only  experts  have  been  tested;  but  your  test  is  not  going  to  be  given 
to  experts.  It  will  be  given  to  students.  Validation  will  tell  you  if  it  will  work  for 
students.  If  your  test  is  really  a  test,  then  masters  of  the  subject  should  score  higher  than 
nonmasters.  If  there  is  no  difference,  the  test  is  really  telling  you  nothing. 

What  then  is  a  master  of  the  subject?  What  is  a  nonmaster?  Should  senior  NCOs 
score  higher  on  your  test  than  junior  NCOs?  If  so,  you  have  a  master/nonmaster 
distinction  right  there.  Should  graduates  of  your  course  generally  score  higher  than 
nongraduates?  Experienced  performers,  higher  than  recruits?  Field  grade  officers,  higher 
than  second  lieutenants?  Any  two  groups  like  these  can  be  termed  master  and  nonmaster. 
To  validate  your  test,  think  of  two  groups— one  group  that  you  expect  to  score  high  and 
one  that  you  expect  to  score  lower.  Of  course,  you  will  need  to  use  your  best  judgment  in 
selecting  the  groups. 

One  of  the  validation  steps  described  below  is  computing  an  "agreement"  index. 
The  agreement  index  refers  to  agreement  between  performance  on  the  test  (right  or 
wrong)  and  whether  the  individual  is  in  the  master  or  nonmaster  group.  For  each  item, 
the  agreement  index  must  be  greater  than  zero.  There  are  many  other  ways  to  validate 
tests,  but  the  agreement  index  is  one  of  the  simplest  and  easiest  to  use.  Follow  these 
steps  to  validate  your  paper-and-pencil  tests: 

1.  Determine  two  groups:  One  group  including  those  that  you  consider  to  be 
masters  of  the  subject;  and  the  other,  that  you  consider  to  be  nonmasters.  Get  other 
subject  matter  experts  to  help  you  make  this  decision.  Your  choice  of  group  members 
should  be  one  that  other  experts  agree  is  realistic. 

2.  Identify  10  to  20  individuals  from  each  group  to  take  your  test.  You  should 
have  a  minimum  of  10  in  each  group  and  a  minimum  of  30  all  together.  (You  do  not  have 
to  have  the  same  number  of  masters  and  nonmasters.) 

3.  Administer  the  test  to  30  or  more  persons  under  the  same  conditions  as  in 
actual  use. 

4.  Score  all  the  tests. 

5.  Looking  at  the  score  sheets,  make  up  a  big  table  like  the  one  in  Figure  9-2. 
Enter  M  for  masters  and  N  for  nonmasters.  Use  P  for  passing  the  Item  and  F  for  failing 
the  Item.  (A  blank  copy  of  the  table  is  provided  for  your  use  on  p.  C-5.) 


70 


Figure  9-2.  Table  for  recording  performance  of  masters  and  non¬ 
masters  in  validating  paper-and-pencil  test  items. 


6.  Compute  the  agreement  index.  For  each  item,  draw  a  four-way  table 
(Figure  9-3)  or  reproduce  and  use  the  form  on  p.  C-6.  Enter  the  number  of  masters  and 
nonmasters  who  passed  and  failed  as  shown  in  Figure  9-3.  Then, 

Agreement  index  =  (A  x  D)  -  (B  x  C). 

The  agreement  index  for  item  1  in  Figure  9-2  would  be  computed  like  this: 


Agreement  index  =  (7x8) 


(11  x  4) 


The  agreement  index  is  12.  Any  level  greater  than  zero  is  acceptable.  In  other  words, 
A  x  D  must  be  greater  than  B  x  C.  (The  bigger  the  agreement  index,  the  better.  An 
agreement  index  of  12  is  nothing  to  shout  about.) 

Note.  When  you  are  computing  an  agreement  index,  you  are  comparing  the 
proportion  of  masters  who  passed  the  item  to  the  proportion  of  nonmasters 
who  passed  the  item.  You  could  use  percentages  instead  if  you  prefer,  but 
the  agreement  index  makes  it  even  simpler. 


PASS 

FAIL 


MASTERS 

NONMASTERS 

A 

B 

7 

11 

C 

D 

4 

8 

Figure  9-3.  Table  for  computing  the  agreement  index  in  validating 
paper-and-pencil  test  items. 


If  the  agreement  index  is  zero  or  negative  (that  is,  if  A  x  D  is  not  greater  than  B 
x  C),  you  should  revise  the  item.  You  can  interview  the  persons  who  took  the  test  to  see 
what  is  wrong  or  you  could  administer  the  expert  tryout  questionnaire  as  before.  If  a  lot 
of  your  items  need  revision  (more  than  one-third),  validate  the  test  again  after  you  have 
revised  it.  Use  a  different  group  of  individuals  than  the  last  time.  Do  not  include  anyone 
who  has  taken  the  test  before.  The  second  time  around,  you  should  have  at  least  7 
masters  and  7  nonmasters  and  a  total  of  at  least  20.  Of  possible,  get  at  least  10  of  each.) 
Perform  the  second  validation  the  same  way  you  did  the  first  one. 

9.2  Tryout  of  Hands-on  Tests 

The  goal  for  trying  out  hands-on  tests  is  to  discover  faulty  performance 
measures  and  instructions  so  that  you  can  revise  your  test  as  needed.  A  faulty 
performance  measure  (1)  is  not  observable,  (2)  is  not  scored  consistently,  or  (3)  does  not 
relate  to  the  job. 

Before  you  begin  the  tryout,  show  your  performance  measures  to  other  subject 
matter  experts,  just  as  you  would  paper-and-pencil  tests.  Ask  them  to  critique  the 
measures  for  clarity,  accuracy,  and  relevance.  When  you  have  changed  the  test  according 
to  these  critiques,  try  out  and  validate  your  hands-on  tests.  The  phases  are  the  same  as 
for  paper-and-pencil  tests,  but  the  steps  in  each  phase  are  very  different. 

9.2.1  Phase  1.  Expert  Tryout  of  Hands-on  Tests 

For  the  expert  tryout  of  a  hands-on  test,  you  will  need  five  experts  (instead 
of  only  three).  Two  of  the  experts  will  act  as  scorers.  The  other  three  will  be  tested.  Get 
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the  best  experts  you  can- -not  recent  graduates,  but  experienced  performers.  Again, 
consider  locating  some  experts  with  whom  you  ordinarily  do  not  work,  because  they  may 
be  more  critical.  Then  follow  these  steps: 

1.  Pick  two  of  the  experts  to  be  scorers.  You  must  have  two  scorers,  not  just 

one. 

2.  Go  over  the  score  sheet  ahead  of  time  with  the  scorers  to  make  sure  they 
understand  what  they  are  supposed  to  do. 

3.  Have  the  two  scorers  together  test  each  of  the  other  three  experts  (three 
tests  in  all).  Lay  out  the  equipment  and  set  up  the  test  station  the  way  it  will  be  when 
you  test  for  record.  Have  just  one  scorer  read  the  directions  to  the  person  being  tested, 
but  have  both  scorers  fill  out  score  sheets  separately.  Have  them  test  only  one  person  at 
a  time. 


4.  Check  for  agreement  between  scorers.  If  the  two  scorers  scored  any  perfor¬ 
mance  measure  differently,  find  out  why.  Talk  over  the  disagreement  with  both  scorers 
and  with  the  person  who  was  tested.  Was  one  scorer  out  of  position  so  he  or  she  could  not 
see  the  performance?  Or  was  there  a  disagreement  about  what  the  performance  measure 
meant?  Find  out  why  the  scorers  disagreed,  then  change  the  instructions  or  the  score 
sheet  to  fix  the  problem. 

5.  Administer  the  hands-on  test  questionnaire  shown  in  Figure  9-4.  (It  is  also 
provided  on  p.  C-7  for  you  to  reproduce  as  needed.) 

6.  Revise  your  test.  If  more  than  one  of  your  five  experts  think  that  the  task 
you  are  testing  is  not  a  job  requirement  or  that  the  test  is  not  fair  or  complete,  revise  the 
test. 


Pay  special  attention  to  any  performance  measures  that  experts  fail.  Find  out  if 
experts  agree  that  their  own  performance  was  wrong.  If  they  do  not  agree  that  they  were 
wrong,  double  check  the  performance  measures  and  the  task  analysis. 

9.2.2  Phase  2.  Validation  of  Hands-on  Tests 


The  purpose  for  validating  hands-on  tests  is  to  see  if  scorers  agree  when  they 
give  the  test  to  students.  Up  to  now,  only  experts  have  been  tested.  Although  experts  do 
not  make  a  lot  of  mistakes,  students  will  score  all  the  way  from  low  to  high.  Giving  the 
test  to  representative  students  is  a  more  realistic  way  to  see  if  scorers  will  really  agree. 

For  validation,  you  will  need  five  students  who  are  familiar  with  the  task  you  are 
testing.  Therefore,  pick  about  ten  and  show  them  a  description  of  the  task.  (You  can  use 
the  task  analysis  as  a  description.)  Then,  give  them  all  this  little  quiz: 

How  well  can  you  do  this  task? 

0.  Not  familiar  with  task  at  all. 

1.  Not  very  well. 

2.  Fairly  well. 

3.  Very  well. 

Try  to  pick  five  examinees  who  marked  02  or  03.  Do  not  include  anyone  who  selected  00. 
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Yes 

No 

1.  Is  the  task  covered  by  the  test  a  job 
requirement? 

If  not,  why  not? 

2.  Does  the  test  provide  a  fair  measure 
of  job  requirements? 

If  not,  what  changes  are  required? 

a. 

Add  performance  measures? 

Which  ones? 

— 

— 

b. 

Delete  performance  measures? 

Which  ones? 

— 

— 

c. 

Modify  performance  measures? 

How? 

— 

— 

d. 

Alter  standards? 

How? 

e. 

Clarify  instructions? 

How? 

— 

— 

f. 

Change  test  site  or  equipment 
requirements? 

How? 

Figure  9-4.  Hands-on  test  questionnaire. 


When  you  have  five  students  to  test,  you  will  need  to  select  four  (that's  right, 
four)  scorers.  Each  scorer  should  have  the  same  MOS  as  the  students  and  be  of  equal  or 
higher  rank.  Give  your  potential  scorers  the  same  little  quiz  you  gave  to  the  students. 
Select  four  scorers  who  picked  answer  #3  (very  well).  Then,  follow  these  steps  to  validate 
your  hands-on  test: 

1.  Hold  a  rehearsal  for  the  scorers.  Have  them  give  each  other  the  hands-on 
test.  You  should  watch  each  scorer  give  the  test  at  least  once. 

2.  Have  all  the  scorers  give  the  test  to  one  student  at  a  time.  All  four  scorers 
will  score  each  student.  The  point  of  validation  is  to  see  if  the  scorers  agree.  That  is  why 
you  have  so  many  scorers. 

3.  Record  scores.  Collect  the  score  sheets  and  build  a  small  table  showing  how 
each  scorer  scored  each  student  on  the  whole  test,  as  shown  below. 
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Scorer 


Student 


1  2  3 

1  P  P  P 

2  P  P  P 

3  P  F  P 

4  P  P  P 

5  F  F  F 


4  P  =  Pass 

P  F  =  Fail 

P 

P 

P 

F 


4.  Compute  "scorer  agreement."  The  minimum  acceptable  level  of  scorer 
agreement  is  80  percent.  Use  the  small  table  to  build  a  bigger  table  (as  shown  below)  to 
show  agreement  and  disagreement  between  pairs  of  scorers.  Since  you  had  four  scorers, 
you  have  six  different  scorer  pairs:  1  <5c  2,  1  &  3,  1  &  4,  2  <Sc  3,  2  &  4,  and  3  &  4: 

Scorer  Pair 


Student 


1&2  1&3  lie  4 


2ic3  2ie4  3ic4 


Total  Total 

Agree(A)  Disagree(D) 


1  A  A  A 

2  A  A  A 

3  D  A  A 

4  A  D  A 

5  A  A  A 


AAA 
AAA 
D  D  A 
DAD 
AAA 


6 

6 

3 

3 

6 


0 

0 

3 

3 

0 


24  6 


To  compute  scorer  agreement,  divide  the  number  of  agreements  by  the  toted  number  of 
agreements  and  disagreements.  In  this  case,  the  scorer  agreement  is  80  percent  (24  *  30 
=  .80  or  80%). 


5.  Revise  the  test  if  scorer  agreement  is  below  80  percent.  If  the  scorer 
agreement  was  below  80  percent,  find  out  why.  If  you  can  fix  the  problem  easily,  do  it. 
Maybe  one  of  the  four  scorers  had  to  stand  so  far  away  that  he  or  she  could  not  observe 
the  examinee,  or  the  problem  may  be  bigger  than  that.  You  may  need  to  reword  some  of 
the  performance  measures.  If  you  change  the  performance  measures  at  all,  go  through 
the  tryout  process  again  to  see  if  your  changes  helped. 

9.3  Evaluation  Implementation 

In  Chapters  2  and  4,  you  saw  how  important  evaluation  Is  in  individualized 
instruction.  Each  of  the  individualized  systems  you  read  about  used  a  lot  of  tests. 
Testing  tells  the  instructor  what  the  student  needs  to  do  next.  You  should  pay  extra 
attention  to  the  way  you  give  tests  in  your  Individualized  system.  This  section  will  discuss 
(1)  giving  tests,  (2)  scoring  tests,  and  (3)  using  test  results. 

9.3.1  Giving  Tests 

In  Individualized  instruction,  the  student  often  takes  more  than  one  test  on  the 
same  topic.  If  you  are  using  paper-and-pencil  tests,  this  means  that  you  probably  need 
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more  than  one  form  of  each  test.  (If  you're  using  hands-on  tests,  one  form  of  each  test  is 
enough.)  This  was  one  of  the  reasons  given  in  Chapter  S  for  building  a  pool  of  items.  The 
item  pool  gives  you  enough  items  to  build  two,  three,  or  more  forms  of  each  test. 

Why  not  give  the  same  test  to  the  same  student  more  than  once?  The  reason,  of 
course,  is  that,  after  the  student  has  seen  the  test  questions,  he  or  she  can  study  how  to 
answer  them.  Giving  the  same  test  again  would  probably  not  sample  the  student's 
knowledge  of  the  whole  subject.  It  would  only  tell  you  how  well  the  student  knew  the 
exact  questions  you  had  put  on  the  test. 

1.  Rotating  forms.  Do  not  use  the  same  test  form  as  the  "first”  test  every 
time.  Rotate  the  forms  of  the  test  so  that  one  student  gets  form  B  the  first  time,  another 
student  gets  form  A,  and  so  on.  There  are  at  least  two  reasons  to  rotate  forms.  It  makes 
your  testing  program  more  "secure."  (Students  do  not  know  which  form  of  the  test  they 
will  get  next.)  Also,  it  evens  out  any  small  differences  in  the  difficulty  of  your  test 
forms. 


2.  Matched  forms.  Ideally,  all  of  your  test  forms  should  be  of  the  same 
difficulty.  Although  you  may  not  be  able  to  make  them  perfectly  equal,  make  them  as 
equal  as  you  can.  To  do  this,  keep  track  of  the  average  score  for  each  form.  (This  is 
another  reason  why  you  need  to  "rotate"  forms.)  After  10  to  20  students  have  taken  each 
form,  compare  the  average  scores.  Then  "juggle"  the  items  among  the  forms  to  even  out 
the  difficulty  level.  Keep  doing  this  as  students  go  through  your  course.  There  is  no 
reason  why  you  cannot  juggle  test  items  around  for  several  months  until  your  test  forms 
are  as  perfectly  matched  as  you  can  make  them. 

3.  Administering  forms.  Each  time  you  give  any  form  of  any  test,  try  to  give 
it  the  same  way.  Remember  that  a  test  is  a  test.  Being  objective  and  not  offering  help  to 
a  student  may  be  hard  sometimes,  but  it  is  the  best  method  of  training.  The  best  way  to 
make  sure  the  test  is  given  the  same  way  every  time  is  to  include  clear,  simple 
instructions  to  the  person  who  will  be  giving  it.  Make  it  clear  that  the  person  giving  the 
test  must  understand  the  instructions  fully,  be  consistent  in  presenting  them,  and  not  give 
special  help  to  any  student. 

9.3.2  Scoring  Tests 

Consistency  and  standardization  should  also  be  your  goals  in  scoring  your  tests. 
Score  each  test  the  same  way  every  time.  This  is  easy  when  the  student  has  scored  very 
high  or  low.  It  gets  harder  to  be  objective  when  the  student  scores  near  the  "borderline." 
However,  if  you  do  not  maintain  test  standards,  your  students  will  soon  know  it.  Then,  it 
will  be  hard  to  explain  why  one  borderline  student  passed  and  another  failed.  Remember 
that,  for  failing  a  test,  a  student  does  not  go  to  Siberia  but  just  "back  to  the  books."  So  do 
not  be  afraid  to  hold  the  line. 

Your  test  keys  will  not  be  perfect  at  first.  No  matter  how  careful  you  were  to 
make  them  correct,  students  will  come  up  with  answers  that  you  had  not  thought  about. 
As  a  result,  you  will  have  to  revise  your  answer  keys. 

Follow  this  rule  when  you  add  or  change  answers  In  the  key:  Do  not  do  it  alone. 
At  least  two  Instructors  should  agree  on  each  addition  or  change  to  prevent  the  "ping 
pong"  effect:  Major  Jones  changes  the  key,  Major  Smith  changes  it  back,  Major  Jones 
changes  the  key,  etc.  You  do  not  need  problems  like  that. 
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9.3.3  Using  Test  Results 


Of  course,  test  results  direct  a  student's  progress  through  your  course.  This  is 
built  into  the  system.  But  you  should  do  more  than  that.  Test  results  can  point  out  where 
a  student  is  having  trouble  and  where  your  course  is  having  trouble. 

Suppose  the  test  shows  that  a  student  does  not  understand  a  particular  part  of 
your  instruction.  What  will  you  do?  Just  send  him  or  her  back  to  study  more?  Of  course 
not.  You  use  the  test  results.  First,  tell  the  student  what  he  or  she  is  doing  wrong.  Then, 
devise  a  short  practice  assignment  or  something  else  to  help  the  student  master  that 
particular  part  of  the  subject. 

Now,  what  if  many  students  are  scoring  low  on  the  same  part  of  the  test?  Again, 
use  your  test  results  and  take  action.  Find  out  whether  the  test  or  the  instruction  is  the 
problem.  (The  problem  is  definitely  not  with  the  students,  since  several  had  the  same 
problem.)  Either  part  of  the  test  is  poorly  built  or  part  of  the  instruction  is  not  clear,  not 
complete,  or  not  correct.  Find  out  why  the  students  are  scoring  low.  Talk  to  them.  Look 
at  the  test.  Look  at  the  instruction.  Use  your  test  results  to  do  something  and  do  not  say 
for  week  after  week  that,  "They  just  don't  seem  to  be  able  to  understand." 
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APPENDIX  A 
STUDY  GUIDE 


How  to  use  the  study  guide.  Read  one  chapter  of  the  handbook 
at  a  time.  During  or  after  reading  the  chapter,  answer  the  study 
questions  here  in  the  guide.  Check  your  answers  and,  if  any  of  your 
answers  are  wrong,  find  out  why  by  rereading  that  section  of  the 
chapter. 
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INTRODUCTION 


Chapter  1  makes  three  big  points: 

1.  An  individualized  system  of  instruction  treats  different  students  in  different 
ways,  usually  according  to  how  they  perform  on  tests. 

2.  In  an  individualized  system,  at  least  one  component  of  instruction  usually 
varies  from  student  to  student.  Chapter  1  shows  you  how  time  and  method  can  vary. 
Chapter  2  will  show  you  how  content  and  achievement  level  can  vary,  too. 

3.  Audiovisual  media  are  often  used  in  individualized  instruction,  but  they  are 
not  a  system.  You  need  more  than  just  media  to  make  an  individualized  system. 

Study  Questions  (Chapter  1) 

1.  True  or  false?  Individualized  instruction  means  that  programmed  texts  are 
used  exclusively. 


a.  True. 

b.  False. 

2.  The  procedures  you  follow  from  the  time  the  student  enters  instruction  until 

he  or  she  graduates  constitute  a _ of  instruction. 

3.  How  is  the  "system  B"  American  history  class  described  in  Chapter  1  more 
individualized  than  the  traditional  system? 

a.  It  used  small-group  instruction. 

b.  It  was  based  on  objectives. 

c.  Certain  students  went  to  extra  classes. 

d.  It  was  graded  pass/fail. 

4.  Name  four  ways  in  which  students  differ  from  each  other  that  can  affect 
their  performance. 


5.  True  or  false?  In  an  individualized  course,  a  student  can  take  as  long  as  he 
or  she  wants  to  finish. 

a.  True. 

b.  False. 

6.  In  a  class  on  amphibious  doctrine,  the  instructor  divided  a  50-m'inute  class 
between  reading  a  case  study  and  viewing  a  tape/slide  lesson.  This  is  an  example  of 
variable 
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7.  Name  at  least  one  limitation  or  shortcoming  of  a  live  presentation  (such  as  a 

lecture)  in  presenting  instruction: _  . 

8.  In  a  perfect,  or  ideal,  individualized  instruction  system,  which  one  of  the 
following  would  be  true? 

a.  Faster  and  slower  students  would  both  get  the  same  amount  of  study 

time. 

b.  Every  student  would  get  a  combination  of  workbook  exercises  and 
individual  tutorial  aid. 

c.  The  use  of  audiovisual  media  would  be  maximized. 

d.  Every  student  would  be  taught  by  the  method  that  matches  his  or  her 
learning  style. 
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Answers  (Chapter  1) 


1.  b.  False. 

2.  System. 

3.  c.  If  you  picked  choice  d,  remember  that  individualized  instruction  is  not 
just  the  use  of  any  particular  method. 

4.  The  text  mentioned:  previous  knowledge,  aptitude  for  learning,  motivation, 
study  habits,  writing  skill,  and  reading  skill.  You  may  have  thought  of  others,  too.  If  you 
are  in  doubt,  show  your  answer  to  an  instructor. 

5.  b.  False.  You  can  combine  variable  time  with  time  limits. 

6.  Method. 

7.  The  text  mentioned  two:  Live  presentations  are  (a)  not  easily  repeated  and 
(b)  not  easily  referred  to.  You  may  have  thought  of  others.  If  you  are  in  doubt,  show  your 
answer  to  an  instructor. 

8.  d.  Unfortunately,  the  "ideal"  system  does  not  exist  yet.  We  do  not  know 
enough  about  the  interaction  between  instructional  method  and  "learning  style." 
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2.0 


APPROACHES  TO  INDIVIDUALIZED  INSTRUCTION 


The  point  of  Chapter  2  is  to  make  you  think  about  individualized  instruction  in 
general  terms  instead  of  in  terms  of  only  one  technique.  This  is  where  you  should  start 
thinking  about  your  own  course  in  terms  of  individualization.  As  you  read  Chapter  2, 
think  about  what  you  want  to  vary  in  your  course:  content,  time,  method,  achievement 
level? 

Study  Questions  (Chapter  2) 

1.  Enter  check  marks  to  classify  the  course  you  teach  now  in  terms  of  the  four 
factors  below. 


Factor  Fixed  Variable 

Content  _  _ 

Time  _  _ 

Method  _  _ 

Achievement  level  _ 


2.  Should  anything  about  your  course  be  changed  to  make  it  easier  to 
individualize  it?  How? 


3.  An  instruction  system  in  which  faster  students  go  on  to  study  electives  is  an 
example  of _ content. 


a.  Fixed. 

b.  Variable. 

4.  A  system  that  uses  self-paced  programmed  texts  only  is  an  example 

of _ method  and _ time. 

a.  Fixed. 

b.  Variable. 

5.  In  a  self-paced  class,  suppose  that  students  have  several  ways  to  study  the 
same  objectives:  study  guides,  texts,  and  tapes.  This  class  has  variable.* 

a.  Time  and  content. 

b.  Time  and  methods. 

c.  Content  and  methods. 

d.  Time,  content,  and  methods. 

6.  You  are  an  instructor  in  a  workshop/tutorial  system.  A  student  has  just 
failed  the  first  test  for  a  particular  lesson.  What  will  you  do  now? 

a.  Assign  a  retest  immediately. 

b.  Provide  individual  tutorial  assistance. 

c.  Have  the  student  read  the  workbook  over  again. 

d.  Send  the  student  to  the  next  lesson. 


7.  In  a  study  guide  system,  the  instructor: 

a.  Never  works  with  the  student  until  after  the  first  lesson  test. 

b.  Lectures  once  each  day. 

c.  Is  available  to  explain  something  that  the  student  does  not  understand. 

d.  Is  essential  to  guide  the  student  through  each  part  of  the  text. 

8.  You  are  an  instructor  in  a  fixed-pace  individualized  system.  In  your  course: 

a.  Students  in  a  class  will  graduate  after  different  lengths  of  time, 
probably  with  different  levels  of  achievement. 

b.  Students  in  a  class  will  all  graduate  at  the  same  time,  probably  with 
different  levels  of  achievement. 

c.  Students  in  a  class  will  all  graduate  at  the  same  time  with  the  same 
level  of  achievement. 

d.  None  of  the  above,  but  students  will  all  spend  the  same  amount  of  time 
in  independent  study. 

9.  You  are  an  instructor  in  a  course  using  a  programmed  text.  One  of  your 
students  has  failed  the  lesson  test  twice.  What  should  you  do? 

a.  Have  the  student  read  the  programmed  text  a  third  time. 

b.  Pass  the  student  on  to  the  next  lesson. 

c.  Fail  the  student  for  the  course. 

d.  Switch  to  a  different  method  of  instruction  for  this  lesson. 

10.  You  are  an  instructor  in  a  worksheet  system.  One  of  your  students  has 
completed  worksheet  set  #2.  What  should  you  do? 

a.  Give  the  lesson  test. 

b.  Assign  worksheet  set  #3. 

c.  Provide  tutorial  assistance. 

d.  Pass  the  student  on  to  the  next  lesson. 

11.  In  the  peer  training  system,  each  student  goes  on  to  the  next  module  when 
he  or  she  has: 

a.  Practiced  the  task  with  a  partner. 

b.  Completed  videotapes,  tape/slides,  etc. 

c.  Given  the  lesson  test  to  his/her  partner. 

d.  Passed  the  lesson  test. 

12.  An  instructor  uses  an  overhead  projector  to  present  information  from  a  text. 
Is  this  individualized  instruction?  Why  or  why  not? 


I.  (Discuss  the  classification  of  your  own  course  with  an  instructor.) 


2.  (If  you  are  in  doubt  about  what  you  could  do  to  make  your  own  course  easier 
to  individualize,  ask  an  instructor.) 

3.  b.  Variable  content. 

4.  a.  Fixed  method, 
b.  Variable  time. 

5.  b. 

6.  b. 

7.  c.  If  you  marked  choice  d,  remember  that  leading  the  student  through  the 
text  is  mostly  the  job  of  the  written  study  guide. 

8.  b.  If  the  course  is  of  fixed  length,  some  students  will  probably  reach 
higher  levels  of  achievement  than  others  in  that  time. 

9.  d.  If  the  student  is  still  having  difficulty,  the  programmed  text  is  not 
likely  to  be  helpful  the  third  time  through. 

10.  a. 

11.  d. 

12.  No,  not  by  itself.  Using  an  audiovisual  medium  like  an  overhead  projector  is 
not  individualization.  But  you  can  use  audiovisual  aids  within  an  individualized  system. 
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3.0  THE  INSTRUCTOR’S  ROLE 
Study  Questions  (Chapter  3) 

1.  True  or  false?  The  main  role  of  the  instructor  in  individualized  instruction 
is  to  give  tests. 


a.  True. 

b.  False. 

2.  One  convenient  time  to  provide  motivation  or  encouragement  is  after 
correcting  a  student's  work.  Name  two  other  convenient  times. 

a.  _ _ 

b.  _ 

3.  True  or  false?  In  an  individualized  course,  the  instructor  is  the  primary 
means  of  motivation  or  encouragement. 

a.  True. 

b.  False. 


it.  True  or  false?  The  most  important  goal  of  diagnosis  is  to  tell  the  student 
the  right  answers. 

a.  True. 

b.  False. 

5.  What  one  thing  must  you  do  to  figure  out  why  a  student  is  having  difficulty 
with  an  exercise? 

a.  Tell  the  student  how  other  students  have  solved  it. 

b.  Ask  the  student  to  practice  more. 


c.  Allow  the  student  to  tell  you  how  he  or  she  did  the  exercise. 

d.  Suggest  that  the  student  reread  the  exercise  while  checking  the 
procedure  in  the  text  or  audiovisual  presentation. 


6.  If  a  student  is  doing  a  series  of  practical  exercises,  when  is  the  best  time  to 
provide  feedback? 


a.  Before  practice. 

b.  During  practice. 

c.  After  practice  is  over. 


7.  True  or  false?  All  the  feedback  a  student  needs  is  generally  provided  by  the 
instruction  materials. 

a.  True. 

b.  False. 


8.  In  what  way  is  counseling  easier  in  an  individualized  system  than  in  a  lock- 
step  system? 


9.  Here  are  four  possible  problems  that  a  student  might  have.  An  instructor 
must  be  able  to  counsel  him  about  which  one? 

a.  He  cannot  understand  the  instruction. 

b.  His  car  payments  are  late. 

c.  He  has  not  been  promoted. 

d.  His  son  is  having  trouble  in  school. 

10.  Which  one  of  the  following  is  an  example  of  direct  instruction? 

a.  An  instructor  counsels  a  student  about  her  low  grades. 

b.  An  instructor  administers  a  hands-on  test. 

c.  An  instructor  demonstrates  a  new  procedure  to  a  student. 

d.  An  instructor  corrects  a  student's  errors  on  a  test. 

11.  One  of  your  students  often  comes  to  class  late.  His  progress  through  the 
lessons  is  slow.  He  has  often  had  to  take  tests  several  times  to  pass.  Which  functions 
should  you  as  his  instructor  perform  in  this  case? 
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Answers  (Chapter  3) 

1.  b.  False.  There  are  many  functions  more  important  than  that  one. 

2.  The  text  names  three  others:  (a)  before  a  student  begins  work,  (b)  when  a 
student  asks  for  an  explanation,  and  (c)  at  the  end  of  a  test  or  class  session.  You  may 
have  thought  of  others.  If  you  are  in  doubt,  show  your  answer  to  an  instructor. 


3. 

a. 

True.  Motivation  is  one 

of  the  things  that  human  beings  generally  do 

better  than 

media. 

4. 

b. 

False. 

5. 

c. 

None  of  the  other  options 

will  tell  you  what  the  problem  is. 

6. 

b. 

7. 

b. 

False. 

8.  In  an  individualized  system,  counseling  is  more  convenient.  The  instructor 
can  counsel  a  student  without  having  to  interrupt  the  whole  class  or  wait  until  after  class 
time. 

9.  a. 

10.  c. 

11.  It  looks  as  though  this  student  problem  needs  all  five:  (a)  encouragement, 
(b)  problem  diagnosis,  (c)  feedback,  (d)  counseling,  and  (e)  direct  instruction. 


I 
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4.0 


DESIGNING  YOUR  INDIVIDUALIZED  SYSTEM 


Chapter  4  reviews  the  basic  parts  of  any  instruction  system,  individualized  or  not, 
because  it  is  so  important  to  include  all  these  parts  in  an  individualized  system.  By  now, 
you  should  be  making  up  your  mind  how  your  own  individualized  system  will  work.  Some 
of  the  questions  below  will  call  on  you  to  say  how  you  plan  to  build  your  own  course. 

Study  Questions  (Chapter  4) 

1.  Review  Question.  State  three  factors  in  addition  to  time  that  you  could 
vary  in  an  individualized  system: 

a.  Time  _ . 


b. 


c. 


d. 


2. 

column  B: 


Match  the  events  in  column  A  with  the  instruction  system  components  in 


A 

B 

Events 

System  Components 

a. _  A  student  asks  a  question  about  a  statement 

(1) 

Presentation. 

in  a  text. 

(2) 

Student  response. 

b. _  The  instructor  tells  a  student  what  she  got 

right  or  wrong  on  a  test. 

(3) 

Feedback. 

c. _  A  student  watches  a  demonstration  on  a 

videotape. 


(4)  Evaluation. 


d.  _  The  instructor  gives  a  lecture. 

e.  _  The  instructor  administers  a  performance  test. 

f.  _  The  instructor  critiques  a  student's  map  overlay. 


g. _  A  student  develops  a  scheme  of  maneuver. 


3.  A  student  asking  the  instructor  questions  during  a  lecture  is  an  example  of: 

a.  Presentation. 

b.  Student  response. 

c.  Feedback. 

d.  Evaluation. 

4.  Answering  practice  exercises  is  an  example  of: 

a.  Presentation. 

b.  Student  response. 

c.  Feedback. 

d.  Evaluation. 


5.  Practice  exercises  themselves  are  examples  of: 


a.  Presentation. 

b.  Student  response. 

c.  Feedback. 

d.  Evaluation. 

6.  Why  will  you  probably  need  more  than  one  form  of  your  tests  in  an 
individualized  course? 


7.  Name  one  way  that  feedback  is  provided  in  the  workbook/tutorial  system  at 
the  Instructional  Management  School: 


8.  What  component  of  instruction  directly  follows  presentation  in  the  basic 
program  of  an  individualized  system? 


9.  In  the  basic  program  of  an  individualized  system,  what  always  follows  a 
student's  response? 


10.  Which  two  components  of  individualized  instruction  are  often  repeated 
several  times  after  a  presentation? 

a.  Evaluation  and  student  response. 

b.  Feedback  and  evaluation. 

c.  Feedback  and  presentation. 

d.  Student  response  and  feedback. 

11.  Name  at  least  two  methods  and/or  media  you  could  use  in  your  own  course 
to  present  content: 


12.  Name  at  least  two  methods  you  could  use  in  your  own  course  to  provide  for 
student  response  after  the  presentation. 


13.  Name  at  least  one  method  you  could  use  in  your  own  course  to  provide 
feedback  after  student  response. 


14.  When  you  are  evaluating  your  students'  performance,  why  do  you  have  to 

sample? 


15.  Name  one  action  you  could  take  in  your  own  course  if  a  student  fails  one  or 
more  lesson  evaluations. 


16.  How  will  you  individualize  your  own  course?  Enter  all  of  your  decisions 
again  in  the  table  below. 


Question 


Your  Decision 


a.  How  will  you  present  instruction  initially? 

b.  How  will  you  provide  for  the  student's 
response? 

c.  How  will  you  provide  feedback  to  the  student's 
responses? 

d.  How  will  you  evaluate  the  student? 

e.  If  the  student  fails  the  first  evaluation, 
what  will  you  provide  in  the  way  of: 

(1)  Additional  presentations? 

(2)  Additional  student  response? 

(3)  Additional  feedback? 

(4)  Additional  evaluation? 

f.  What  will  the  system  do  if  the  student 
"never"  passes  the  test? 


17.  Now,  classify  the  system  you  have  in  mind  for  your  own  course,  according  to 
these  four  factors: 


Fixed  Variable 

a.  Content.  _  _ 

b.  Time.  _  _ _ 

c.  Method.  _  _ 

d.  Achievement  level.  _ 
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1.  a.  Time. 

b.  Content. 

c.  Method. 

d.  Achievement  level. 

2.  a.  (2)  Student  response. 

b.  (3)  Feedback. 

c.  (1)  Presentation. 

d.  (1)  Presentation. 

e.  (4)  Evaluation. 

f.  (3)  Feedback. 

g.  (2)  Student  response. 

3.  b. 

4.  c. 

5.  b. 

6.  Because  individualization  calls  for  repeated  testing.  If  a  student  fails  a  test 
one  time,  you  probably  will  not  want  to  ask  the  very  same  questions  the  second  time. 

7.  Either  one  of  these  is  correct:  (a)  answers  to  practice  exercises  in 
workbooks  or  (b)  instructor  feedback. 

8.  Student  response. 

9.  Feedback. 

10.  d. 

11.  The  text  mentions  the  following  ten  ways  of  presenting  content:  printed 
text  or  workbook,  videotape,  tape/slide,  motion  picture,  closed-circuit  television,  com¬ 
puter,  lecture/demonstration,  conference  group,  teleconference,  and  programmed  text. 
The  methods  you  picked  for  your  course  may  be  among  these  or  you  may  have  thought  of 
others. 

12.  The  text  lists  the  following  ways  to  provide  for  student  response:  study 
questions,  quizzes,  problems/case  histories,  drill-type  exercises,  and  creation  of  a 
product.  The  methods  you  picked  for  your  course  may  be  among  these  or  you  may  have 
thought  of  others. 

13.  The  text  lists  the  following  ways  to  provide  feedback:  answer  keys,  model 
answers  ("school  solution"),  and  instructor  critique.  The  method(s)  you  picked  for  your 
own  course  may  be  among  these  or  you  may  have  thought  of  others. 

14.  Because  you  usually  cannot  put  everything  on  a  test.  You  cannot  have  the 
student  demonstrate  all  performance  or  all  knowledge. 


Jl 


15.  The  text  suggests  these  actions  if  a  student  fails  one  or  more  lesson 
evaluations:  Restudy  primary  presentation,  introduce  and/or  restudy  alternate  (sec¬ 
ondary)  presentation  method,  and  receive  tutorial  aid  from  instructor  or  another  student. 
The  way(s)  you  picked  for  your  own  course  may  be  among  these  or  you  may  have  thought 
of  others. 

16.  (Show  your  completed  table  of  decisions  to  an  instructor.) 

17.  (Show  your  completed  classification  scheme  to  an  instructor.) 
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5.0  QUALITY  CONTROL  AND  MANAGEMENT 

Remember  that  the  quality-control  methods  in  Chapter  5  are  based  on  the 
instructional  quality  inventory  (IQI)  (Ellis,  Wulfeck,  Sc  Fredericks,  1979).  There  should  be 
copies  of  the  IQI  in  your  IMS  classroom.  The  IQI  covers  the  subject  of  quality  control  in 
detail  with  examples  and  lots  of  practice. 

Study  Questions  (Chapter  5) 

1.  The  IQI  gives  you  three  rules  for  effectiveness  in  the  presentation  of 

content: 

a.  Separate  important  points. 

b.  Identify  important  points. 

c.  Provide  help  to  remember  important  points. 

Which  two  of  these  three  rules  have  been  followed  in  the  passage  below? 

In  constructing  a  Big  Cheese  sandwich,  always  place  the  second 
slice  of  cheese  between  the  top  burger  patty  and  the  middle  bun. 

Note:  Never  spread  mayonnaise  on  top  of  the  cheese. 

a. 


b.  _ 

2.  Give  two  examples  of  labels  you  can  use  to  identify  important  points: 
a. 


b. 


3.  Make  up  a  "memory  trick"  (mnemonic)  to  remember  the  four  types  of 
amphibious  operations: 

a.  Assault. 

b.  Withdrawal. 

c.  Demonstration. 

d.  Raid. 


4.  From  memory,  state  one  of  the  rules  for  adequate  examples. 


5.  State  one  of  the  rules  for  adequate  practice  items. 
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6.  Classify  the  following  objectives  "List  the  two  branches  of  an  infantry 

division." 

a.  Use  principle  (aided). 

b.  Remember  procedure. 

c.  Use  category  (unaided). 

d.  Remember  fact. 

7.  Classify  the  following  test  item: 

For  the  3rd  Calendar  Inspection  begun  on  an  aircraft  on  a 
given  day,  the  Job  Control  Sequence  Number  would  be 

(1) 003 

(2)  030 

(3)  B03 

(4)  COO 

a.  Use  procedure  (unaided). 

b.  Remember  procedure. 

c.  Use  principle  (aided). 

d.  Remember  category. 

8.  Look  at  each  of  the  student  progress  cards  (Figure  A-1--  A-10)  on  the 
following  pages.  For  each  student,  enter  the  letter  (below)  representing  the  action  the 
instructor  should  take.  For  this  exercise,  today's  date  is  Thursday,  19  November  and  the 
passing  criterion  is  90  percent. 


(1)  Greenberg 

a. 

Administer  test  form  A. 

(2)  Ralston 

b. 

Administer  test  form  B. 

(3)  Davidson 

c. 

Administer  test  form  C. 

(4)  Kahl 

d. 

Assign  worksheets  Part  1. 

(5)  Fox 

• 

e. 

Assign  worksheets  Part  2. 

(6)  Lathrop 

f. 

Check  regularly  to  see  if 

(7)  Brzinski 

student  can  use  assistance. 

(8)  Johnson 

g- 

Assign  student  to  next  lesson. 

(9)  Green 

(10)  Cohn 

9.  Next,  make  up  a  student  progress  card  for  your  own  course.  You  will  need 
lots  of  scratch  paper.  When  you  are  finished,  explain  to  your  instructor  how  a  student 
would  progress  through  your  course.  When  your  instructor  is  satisfied  with  your  work, 
have  him  or  her  sign  off  here: 


Instructor  Sign-off  on  Student 
Progress  Card 
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MODULE 


me  rhj  &  (/ ) 


Figure  A- 1.  Student  progress  card  1. 
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>g  /2/U^STa  V _ Qkl _  _ MODULE  ■&-  "7 


Figure  A-2.  Student  progress  card  2. 
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Figure  A-3.  Student  progress  card  3. 


KAHE  k.AHi-  C<f) _ WMU^Z-Z 


Figure  A-4.  Student  progress  card  k. 
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NAME 


Csl 


MODULE 


Figure  A-5.  Student  progress  card  5. 


NAME  h&nLE&fL  Cc.  2  _ MODULE  ^  V 


Figure  A-6.  Student  progress  card  6. 
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HAME_  Bft7.lt/5KI  Cl)  _ MODULE  7 


Figure  A-7.  Student  progress  card  7. 
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Figure  A-8.  Student  progress  card  8 


NAM£_  g /€/$£//  Cll  _  MODULE  ^  7 


Figure  A-9.  Student  progress  card  9. 
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Answers  (Chapter  5) 


1.  a.  Separate  important  points, 

b.  Identify  important  points. 

The  important  point  about  the  mayonnaise  is  separated  by  being  indented  and 
identified  by  the  word  note. 

2.  The  text  suggests  six: 

a.  Key  point. 

b.  Rule. 

c.  Definition. 

d.  Procedure  for _ . 

e.  Principle  of _ . 

f.  Main  idea. 

These  are  just  suggestions.  You  may  have  thought  of  others. 

3.  One  example  of  a  mnemonic  to  remember  the  four  types  of  amphibious 
operations  is:  "A  wicked  dentist  returns"  (AWDR). 

k.  The  1QI  gives  six: 

a.  Examples  must  be  separated  and  identified. 

b.  Examples  must  include  some  type  of  help. 

c.  Examples  should  range  from  easy  to  hard. 

d.  Examples  should  represent  the  job  the  student  will  do  after  training. 

e.  There  should  be  enough  examples  to  cover  the  content  area  adequately. 

f.  Examples  should  clearly  show  why  common  errors  are  wrong. 

The  first  two  rules  are  the  same  as  those  for  presentations.  You  do  not  have  to 
memorize  these  rules,  but  the  more  of  them  you  keep  in  mind  when  you  write  your 
instruction,  the  less  you  will  have  to  change  later. 

5.  The  IQI  gives  seven: 

a.  The  practice  section  must  be  separate  and  identified. 

b.  The  practice  items  must  be  free  of  hints  that  will  not  be  present  in  the 
test  or  on  the  job. 

c.  The  practice  items  should  have  the  same  format  as  the  format  of  test 

items. 

d.  The  practice  items  should  range  from  easy  to  hard. 

e.  The  practice  items  should  be  typical  of  the  job  to  be  performed  after 

training. 

f.  The  practice  items  should  include  the  opportunity  for  common  errors. 

g.  The  practice  items  must  be  followed  by  feedback. 
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6.  d.  Remember  fact. 

7.  a.  Use  procedure  (unaided).  This  is  a  use  item  because  the  student  has  to 
apply  the  general  procedure  to  figure  out  the  number  for  a  particular  inspection  (the  third 
one  that  day).  It  is  unaided  because  statement  of  the  rule  is  not  stated  for  the  student. 

8.  a  (1)  Greenberg.  Administer  test  form  A.  Greenberg  finished  her  work¬ 
sheets  yesterday.  She  is  ready  for  the  test  again. 

f  (2)  Ralston.  Check  regularly  to  see  if  Ralston  can  use  assistance.  He 
is  working  on  worksheets,  part  2. 

c  (3)  Davidson.  Administer  test  form  C.  As  Davidson  finished  work¬ 
sheets,  part  2,  yesterday,  he  is  ready  for  the  test. 

f  (4)  Kahl.  Kahl  is  in  the  same  situation  as  Ralston — working  on  work¬ 
sheets,  part  2.  Check  regularly  to  see  if  he  needs  help. 

f  (5)  Fox.  Fox  is  working  on  worksheets,  part  1.  Check  regularly  to  see 
if  she  needs  help,  too. 

r  (6)  Lathrop.  He  passed  (90%)  the  test  for  module  #7  yesterday. 
Assign  him  to  the  next  lesson. 

\ 

e  (7)  Brzinski.  Since  she  failed  to  master  the  lesson  test  just  now 
(today,  19  November),  assign  worksheets,  part  2.  (Of  course,  if  you  had  already  assigned 
part  2,  your  answer  would  be  to  check  regularly  to  see  if  she  needs  help.) 

b  (8)  Johnson.  He  is  ready  for  the  test  again,  because  he  has  just 
finished  worksheets,  part  1. 

R  (9)  Green.  She  has  mastered  this  lesson;  assign  her  to  the  next  lesson. 

f  (10)  Cohn.  Cohn  is  working  on  worksheets,  part  1.  Keep  an  eye  on  him 
to  see  if  he  needs  help. 

9.  When  you  have  finished  your  student  progress  card  for  your  own  course, 
explain  it  to  your  instructor.  Then,  have  him  or  her  sign  off  in  the  space  provided  in  your 
study  guide. 
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THE  ROLE  OF  EVALUATION 


Evaluation  is  part  of  your  instructional  package  because  individualization  makes 
large  demands  on  evaluation.  In  individualized  instruction,  evaluation  is  a  tool  for 
affecting  the  student's  performance.  Tests  are  an  active  part  of  the  instruction  process, 
not  just  its  end.  Therefore,  your  evaluation  system  has  to  give  the  instructor  and  the 
student  up-to-the-minute  information  on  what  to  do  next. 

Study  Questions  (Chapter  6) 

1.  State  at  least  two  reasons  to  evaluate  the  individual  performance  of 

students. 


a. 


b. 


2.  Why  is  it  important  for  test  results  to  be  available  quickly  in  individualized 
instruction? 

a.  It  is  important  for  instructors  to  find  out  quickly  who  is  the  best 

student. 

b.  Test  results  help  determine  what  a  student  studies  next. 

c.  Individualized  courses  are  shorter  than  traditional  courses. 

d.  Tests  themselves  are  usually  shorter  in  individualized  courses. 
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Answers  (Chapter  6) 

1.  The  text  mentions  four  reasons: 

a.  To  see  if  training  objectives  (of  skill  and  knowledge)  have  been  met. 

b.  To  see  what  skills  and  knowledge  a  student  still  needs  to  acquire. 

c.  To  determine  eligibility  for  special  duty,  special  pay,  etc. 

d.  To  evaluate  units  as  a  whole. 

You  may  have  thought  of  other  reasons  to  evaluate  individuals  (like  initial 
selection  into  the  service  or  classification  to  a  job).  If  you  are  in  doubt,  show  your  answer 
to  an  instructor. 


7.0  PLANNING  A  TEST 

Planning  the  test  is  the  most  important  phase  of  the  test  development  process. 
Chapter  7  aims  to  make  you  think  about  what  you  are  trying  to  measure  before  you 
"plunge  in." 

Study  Questions  (Chapter  7) 

1.  Which  is  the  best  definition  of  the  performance  or  content  domain  of  a  test? 

a.  The  set  of  all  questions  on  a  test. 

b.  All  of  the  knowledge  and  skill  in  a  job. 

c.  The  hands-on  portion  of  a  skills  test. 

d.  The  knowledge  and  skills  that  you  want  the  test  to  measure. 

2.  True  or  false?  The  performance  domain  refers  to  job  performance,  never  to 
school  performance. 

a.  True. 

b.  False. 

3.  You  want  to  evaluate  each  student's  ability  to  lead  a  battalion  landing  team 
in  an  amphibious  assault,  but  you  do  not  have  time  or  facilities  for  each  student  to 
participate  in  a  landing.  How  could  you  test  to  see  if  your  students  have  this  ability? 


4.  You  are  a  member  of  an  evaluation  team  in  the  field.  You  are  testing  25 
Marines  at  random  to  see  how  well  they  can  apply  first  aid  for  burns.  What  is  the  most 
reasonable  explanation  for  such  a  testing  program? 

a.  You  are  interested  only  in  how  well  those  25  Marines  can  perform 

today. 

b.  You  are  interested  only  in  finding  out  who  knows  the  most  about  burns. 

c.  You  are  interested  in  how  Marines  in  general  will  be  able  to  apply  first 
aid  if  it  is  needed. 


d.  You  are  interested  only  in  finding  out  what  those  25  Marines  learned  in 

training. 

5.  Of  the  following  five  methods  for  picking  test  items,  which  would  usually  be 
the  simplest  to  carry  out? 

a.  Random  sampling. 

b.  Generality  of  content. 

c.  Frequency. 

d.  Performance  characteristics. 

e.  Criticality. 
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6.  Generality  of  content  can  be  used  as  a  sampling  strategy  when: 

a.  You  know  that  frequently  performed  tasks  are  important. 

b.  Information  learned  and  job  performance  are  not  related. 

c.  You  can  determine  the  criticality  of  knowledge. 

d.  You  can  identify  commonalities  among  the  tasks  that  make  up  perfor¬ 
mance. 

7.  What  is  one  reason  to  represent  frequently  performed  tasks  more  heavily  on 

a  test? 


8.  You  are  planning  a  test.  You  have  decided  to  select  the  tasks  that  take  the 
longest  time  to  learn.  This  is  an  example  of  sampling  on  the  basis  of: 

a.  Performance  characteristics. 

b.  Frequency  of  performance. 

c.  Criticality. 

d.  Generality  of  content. 

9.  You  are  planning  a  test.  You  have  decided  to  represent  the  tasks  that  are 
most  important  in  terms  of  logistics.  This  is  an  example  of  sampling  on  the  basis  of: 

a.  Performance  characteristics. 

b.  Frequency  of  performance. 

c.  Criticality. 

d.  Generality  of  content. 

10.  The  form  of  test  that  is  best  for  evaluating  students  depends  mainly  on: 

a.  The  interests  and  aptitudes  of  the  students  in  the  class. 

b.  The  kind  of  knowledge  or  skill  you  are  measuring. 

c.  Whether  you  are  sampling  on  the  basis  of  criticality  or  frequency. 

d.  Whether  you  are  evaluating  for  remediation  purposes  or  diagnosis. 

11.  Select  the  best  definition  of  a  procedure: 

a.  A  task  that  calls  for  a  lot  of  judgment. 

b.  A  task  that  calls  for  quick  reflexes  and  strength. 

c.  A  task  that  calls  for  following  a  set  of  directions. 

d.  A  task  that  calls  for  remembering  when  to  apply  one  rule  and  when  to 
apply  a  different  rule. 
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12.  Enter  either  A  or  B  below  to  show  whether  each  of  the  following  involves 
mostly  application  of  a  procedure  (A)  or  a  motor/perceptual  skill  (B). 

a.  _  Pitching  horseshoes. 

b.  _  Hitting  a  target  with  a  hand  grenade. 

c.  _  Preparing  DD  Form  4444. 

d.  _  Starting  a  1/4- ton  vehicle. 

e.  _  Performing  preventive  maintenance  on  a  radio. 

f.  _  Performing  a  parachute  landing  roll. 

13.  Check  the  statements  below  that  involve  mostly  the  application  of  a  rule  or 
principle. 

a.  _  Filling  out  a  payment  voucher. 

b.  _  Writing  test  items. 

c.  _  Disassembling  a  rifle. 

d.  _  Changing  all  spark  plugs  in  a  vehicle. 

e.  _  Selecting  the  proper  munition  to  employ  in  a  tactical  situation. 

14.  Which  one  of  the  following  is  most  characteristic  of  problem  solving  and 
decision  making? 

a.  The  performer  must  follow  very  closely  the  rules  given  to  him. 

b.  The  performer  must  come  up  with  rules  of  his  own. 

c.  The  performer  must  remember  many  rules  at  once. 

d.  The  performer  must  follow  a  detailed  procedure. 

15.  Match  the  characteristics  in  Column  A  with  the  form  of  test  in  Column  B. 


A 

B 

Characteristics 

Test  Form 

a. _  Can  maximize  the  amount 

of  job  information  in  a  test. 

(1) 

Paper-and-pencil . 

(2) 

Hands-on 

b. _  More  variation  in  scoring. 

performance. 

c. _  More  likely  to  test  unimportant  knowledge. 


d.  _  Automatically  represents  all  knowledge 

needed  to  perform  the  task. 

e.  _  Time  consuming  to  administer. 
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Answers  (Chapter  7) 


1.  d. 

2.  b.  False. 

3.  The  text  suggests  two  possible  ways: 

a.  Test  part  of  the  student's  proficiency.  For  example,  you  could  develop  a 
simulation  of  one  aspect  of  battlefield  communications  and  test  each  student  on  that 
ability. 


b.  Test  part  of  the  student's  knowledge.  For  example,  you  could  develop 
tests  of  knowledge  of  amphibious  planning,  command  and  staff  actions,  offensive 
operations,  etc. 

4.  c. 

5.  a. 

6.  d. 

7.  The  text  gives  one  reason  to  represent  frequent  tasks  more  heavily: 
Frequent  tasks  are  obviously  important,  because  they  are  done  so  often.  You  may  have 
thought  of  another.  If  you  are  in  doubt,  show  your  answer  to  an  instructor. 


8.  a. 

9.  c. 

10.  b. 


11.  c. 


12.  a.  B  Pitching  horseshoes  (skill). 

b.  B  Hitting  a  target  with  a  hand  grenade  (skill). 

c.  A  Preparing  DD  Form  4444  (procedure). 

d.  A  Starting  a  1 /4-ton  vehicle  (procedure). 

e.  A  Performing  preventive  maintenance  on  a  radio  (procedure). 

f.  B  Performing  a  parachute  landing  roll  (skill). 

13.  a.  _  Filling  out  a  payment  voucher. 

b.  X  Writing  test  items. 

c.  _  Disassembling  a  rifle. 

d.  _  Changing  all  the  spark  plugs  in  a  vehicle. 

e.  X  Selecting  the  proper  munition  to  employ  in  a  tactical  situation. 

14.  b. 


15.  a.  (1)  (Paper-and-pencil) 

b.  (2)  (Hands-on) 

c.  (1)  (Paper-and-pencil) 

d.  (2)  (Hands-on) 

e.  (2)  (Hands-on) 
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8.0  CONSTRUCTING  A  TEST 

Chapter  8  stresses  that,  for  paper-and-pencil  tests,  you  should  start  by  developing 
a  whole  pool  of  items  rather  than  just  a  single  test.  The  best  of  the  four  types  of  items 
covered  in  the  text  is  probably  multiple-choice.  Each  of  the  others,  however,  has  its  own 
characteristics  you  should  learn  about. 

The  text  points  out  that  performance  tests  must  be  based  on  task  analysis.  Task 
analysis  is  taught  at  IMS  and  is  not  covered  in  this  text.  If  you  have  not  done  the  task 
analysis,  you  have  nothing  on  which  to  base  a  performance  test.  In  fact,  paper-and-pencil 
tests  are  really  based  on  task  analysis  too,  because  that  is  where  training  objectives  come 
from  in  the  first  place. 

Study  Questions  (Chapter  8) 

1.  State  one  reason  why  even  "performance"  tests  are  not  actually  measures  of 
performance. 


2.  State  one  reason  why  you  should  build  a  "pool"  of  test  items  before  you 
construct  a  paper-and-pencil  test. 


3.  One  reason  that  free  response  test  items  are  hard  to  score  objectively  is 

that: 

a.  Free  response  items  have  one-word  answers. 

b.  Students  may  use  different  words  for  the  same  thing. 

c.  The  student  has  to  know  exactly  what  answer  to  give. 

d.  Statements  cannot  be  taken  directly  from  text. 

4.  What  is  the  problem  with  using  qualifying  words  such  as  never,  always, 
exclusively,  and  sometimes  in  a  true/false  item? 

a.  They  make  statements  always  true  or  always  false. 

b.  They  make  the  test  item  unreliable. 

c.  Some  students  will  not  understand  your  meaning. 

d.  Some  students  "cue"  on  these  words  to  figure  oui  the  right  answer 
easily. 
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5.  From  the  choices  given  below,  choose  the  one  that  states  what  is  wrong  with 
this  matching  item: 

Match  each  weapon  in  column  A  to  the  type  of  unit  that  predominantly 
uses  it  in  column  B. 


A  B 


(b)(1) 

Rifle 

(a) 

Artillery 

(d)  (2) 

Sidewinder 

(b) 

Infantry 

(a)  (3) 

8"  howitzer 

(c) 

Headquarters 

(d) 

Aviation 

a.  The  lists  of  options  are  too  short. 

b.  The  elements  in  column  A  are  not  all  of  the  same  category. 

c.  The  elements  in  column  B  are  not  all  of  the  same  category. 

d.  The  basis  for  matching  is  not  clear. 

6.  State  (below)  at  least  two  rules  to  follow  in  writing  multiple-choice  items: 
a.  _ _ 


b. 


7.  You  should  have  your  test  items  reviewed  by  another  subject  matter  expert 
to  make  sure  that  they  are  clear  and  that  the  paper-and-pencil  format  is  appropriate  to 
what  you  are  trying  to  measure.  What  is  a  third  purpose  of  test  review? 


8.  The  two  general  types  of  scoring  for  performance  tests  are  product-scoring 
and  process-scoring.  Which  type  is  represented  by  the  performance  test  below? 

a.  Product-scored. 

b.  Process-scored. 


GO  NO-GCI 


A.  Landed  on  balls  of  feet  with  knees  bent  slightly 

and  feet  together. 

B.  Upon  contact  with  mat: 

Lowered  chin  to  chest. 


Brought  hands  up  in  front  of  head  with  elbows 
in  front  of  chest. 


Bent  and  twisted  torso  sharply  to  the  right, 
forcing  body  into  an  arc. 

C.  Roiled  in  the  direction  of  drift  (left)  without 

hesitating  on  balls  of  feet. 

D.  Touched  left  calf,  left  thigh,  left  buttock,  and 

fleshy  muscles  in  the  left  side  of  the  back  to 
mat  in  sequence. 

E.  Brought  feet  around  to  right  into  line  of  drift. 

F.  Maintained  tension  in  neck  throughout  fall. 


9.  This  performance  measure  has  a  basic  flaw.  State  what  should  be  done  to 

fix  it. 


Surface  must  be  free  of  grease  and  paint  must  be  applied  evenly. 

_ GO  _ NO-GO 


10.  Two  possible  aids  to  scoring  hands-on  tests  are  templates  and  diagrams. 
Name  two  other  possible  aids. 


a. 


b. 
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11.  There  are  at  least  four  things  you  need  to  tell  the  student  before  a  hands-on 
test.  Fill  in  the  missing  two  below: 

a.  State  the  job  situation. 

b.  _ 


c. 


d.  State  what  help  the  student  will  get. 


12.  What  is  the  maximum  number  of  students  that  can  be  tested  at  one  time  in  a 
process-scored  hands-on  test? 


a.  1. 

b.  2. 

c.  3. 

d.  4. 
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Answers  (Chapter  8) 


i.  The  text  mentions  two: 

a.  The  person  taking  a  "performance"  test  knows  it  is  a  test,  not  a  real 
work  situation. 

b.  A  performance  test  cannot  include  everything  that  occurs  in  real  job 

performance. 


You  may  have  thought  of  other  ways,  too,  such  as  the  fact  that  a 
performance  test  is  given  under  the  same  conditions  every  time,  while  job  performance 
may  occur  under  many  conditions.  If  you  are  in  doubt,  show  your  answer  to  an  instructor. 

2.  The  text  gives  three  reasons: 

a.  An  item  pool  gives  you  enough  items  to  diagnose  what  a  student  is  doing 

wrong. 

b.  An  item  pool  gives  you  items  for  building  more  than  one  test  form. 

c.  An  item  pool  gives  you  enough  extra  items  so  you  can  discard  poor 
items  during  tryouts  and  still  have  enough  left  for  a  test. 

You  may  have  thought  of  other  good  reasons  to  build  an  item  pool,  such  as 
"juggling"  items  from  test  to  test  to  improve  test  security.  If  you  are  in  doubt,  show  your 
answer  to  an  instructor. 

3.  b.  The  instructor  must  interpret  the  answers  when  students  use  different 
words  for  the  same  thing. 


4.  d. 


5.  c.  The  "extra"  element  in  column  B--headquarters--is  obviously  not  a 
combat  arm  as  are  the  others. 

6.  The  text  mentions  five: 

a.  Several  persons  should  agree  about  the  correct  answer. 

b.  Do  not  use  negatively  stated  items. 

c.  Use  "believable"  distractors. 

d.  Do  not  use  overlapping  responses. 

e.  Randomize  the  location  of  the  correct  response. 

7.  To  ensure  that  the  item  is  relevant  to  the  job. 

S.  b.  This  is  a  set  of  process-scored  measures,  because  the  scorer  is  watching 
to  see  how  each  step  of  the  task  is  performed. 
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9.  This  performance  measure  should  be  split  into  two  measures,  because  it  now 
has  two  elements  to  check; 

a.  Was  the  surface  clean? 

b.  Was  the  paint  even? 

The  scorer  should  be  able  to  indicate  easily  if  the  student  performed  one 
action  but  not  the  other. 

10.  The  text  mentions  three  others: 

a.  Overlay. 

b.  Photograph  of  correct  product. 

c.  Photograph  of  faults. 

You  may  have  thought  of  others,  such  as  a  model  of  the  correct 
performance.  If  you  are  in  doubt,  show  your  answer  to  an  instructor. 

11.  b.  State  what  the  student  is  to  do. 

c.  State  the  time  and/or  accuracy  limits. 

12.  Only  one  student  can  be  tested  at  a  time  in  process-scoring,  because 
the  scorer  has  .o  watch  him  or  her  the  whole  time. 
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9.0  TEST  TRYOUT  AND  IMPLEMENTATION 

The  procedures  for  trying  out  your  tests  are  different  for  paper-and-pencil  tests 
and  hands-on  tests.  The  first  exercise  gives  you  practice  in  the  expert  tryout  phase  for  a 
paper-and- pencil  test.  Remember  that  you  are  trying  to  locate  faulty  test  items. 

Study  Questions  (Chapter  9.0) 

Paper-and-pencil  Tests:  Expert  Tryout 

Practice  Situation.  You  are  trying  out  a  paper-and-pencil  test  with  three  experts: 
Murtaugh,  Alston,  and  Stengal.  Table  A-l  presents  the  expert  tryout  questionnaire  and  a 
summary  of  answers.  Two  of  the  experts  (Murtaugh  and  Alston)  passed  all  the  items.  One 
expert  (Stengal)  failed  one  item  (#5). 

Table  A-2  presents  suggested  actions  you  could  take  if  experts  find  a  test  item  is 
not  acceptable.  The  actions  in  this  table  are  keyed  to  the  eight  questions  on  the  expert 
tryout  questionnaire  presented  in  Table  A-l.  (This  table  is  only  a  guide.  When  you  try  out 
your  paper-and-pencil  tests  with  experts,  you  still  have  to  ask  them  questions  about  the 
test.) 

Your  Task.  Look  over  Table  A-l.  Then,  use  the  suggestions  on  Table  A-2  to  help 
you  answer  the  questions  below. 

1.  Are  any  items  questionable?  Which  ones? 


2.  Should  you  interview  anyone?  Whom? 


3.  Do  you  think  you  will  need  to  revise  the  item(s)?  How? 
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Table  A-l 

Expert  Tryout  Questionnaire  with  Experts'  Answers 


Expert  Tryout  Questionnaire 


1.  Do  you  think  you  answered  each  item 
correctly? 

A.  Yes.  B.  No.  C.  Not  sure. 

(Go  to  4)  (Go  to  2)  (Go  to  3) 

2.  Which  items  do  you  think  you  did  not 
answer  correctly? 

3.  Which  items  are  you  unsure  about? 

4.  Is  this  test  realistic?  (Does  it 
reflect  how  the  job  is  done?) 

A.  Yes.  B.  No.  C.  Not  sure. 

5.  Were  the  words  or  phrases  in  the  test 
easy  to  understand?  (If  "No,"  indicate 
in  the  space  below  the  words  or  phrases 
that  gave  you  trouble.) 

A.  Yes.  B.  No.  C.  Not  sure. 

6.  Are  all  of  the  alternatives  to  each 
item  believable?  (If  "No,"  indicate 
the  alternatives  that  are  not  believ¬ 
able.) 

A.  Yes.  B.  No.  C.  Not  sure. 


7.  Are  all  of  the  items  relevant  to  the 
job?  (If  "No"  or  "Not  sure,"  indicate 
which  items  should  be  deleted.) 

A.  Yes  B.  No.  C.  Not  sure. 

8.  As  the  test  is  now  written,  do  you  think 
it  is  a  fair  measure  of  a  Marine's  ability 
to  perform  on  the  job? 

A.  Yes.  B.  No.  C.  Not  sure. 


_ Answers  Given  by  Expert 

Murtaugh  Alston  Stengal 


A  C  A 


3  &  5 

A  A  B 


A  A 


B 

Item  5,  "Acti¬ 
vate  secondary 
firing  device." 


A  A  B 

Item  5,  the 
gunner  would 
not  really  be 
able  to  reach 
the  charging 
handle. 

ABA 
Item  3,  the 
rammer  is 
not  used  in 
combat. 

A  A  C 


Table  A-2 

Suggested  Actions  if  Test  is  Not  Acceptable  to  Experts 


Expert's 

Question  No. 

Answer 

Possible  Actions 

1 

B  or  C 

a. 

Check  that  answer  is  correct. 

b. 

Check  "wrong"  alternatives  are  incorrect. 

c. 

Rewrite  stem. 

d. 

Rewrite  situation. 

e. 

Rewrite  alternatives. 

2 

N/A 

3 

N/A 

H 

B 

a. 

Check  that  answer  is  correct. 

b. 

Add  illustration. 

c. 

Rewrite  situation. 

d. 

Consider  hands-on  testing. 

5 

B 

a. 

Rewrite  entire  item. 

b. 

Add  illustration. 

6 

B 

a. 

Change  unbelievable  alternatives. 

7 

B 

a. 

Delete  entire  item. 

b. 

Rewrite  situation. 

8 

B  or  C 

a. 

Probe  for  more  information  on  preceding 
questions. 

Note.  See  Table  A-l  for  questions  with  answer  keys. 


Paper-and-pencil  Test:  Validation 

The  next  practice  exercise  is  on  the  validation  phase  of  a  paper-and-pencil  test. 
Remember  that  you  are  trying  to  find  test  items  that  do  not  separate  masters  from 
nonmasters.  Simply  compute  the  agreement  index  for  items  1  through  3,  based  on  the  test 
item  scores  on  the  Figure  A-l  1.  (You  will  need  to  make  a  4-way  table  for  each  item,  like 
that  shown  in  Figure  9-3.) 

4.  Agreement  index  for  item  it 1  = _ . 

5.  Agreement  index  for  item  it 2- _ . 

6.  Agreement  index  for  item  it 3  = _ . 

7.  Which  item(s),  (if  any),  must  be  revised  because  the  agreement  index  was  to 

low? 


a.  Item  1  only. 

b.  Item  2  only. 

c.  Item  3  only. 

d.  Items  1  &  2. 

e.  Items  2  &  3. 

f.  Items  3  &  4. 

g.  All  items. 

h.  No  items. 


Hands-on  Test:  Expert  Tryout 

Next,  you  will  practice  locating  defects  in  hands-on  tests  on  the  basis  of  the 
expert  tryout.  Remember  that  you  are  trying  to  find  faulty  performance  measures. 

You  would  use  the  following  procedure  for  the  expert  tryout  phase  of  a  hands-on 
test:  (1)  locate  five  experts,  (2)  pick  two  to  score  the  test,  (3)  have  them  score  the 
remaining  three  experts,  and  (4)  administer  a  questionnaire.  This  resulting  information 
will  indicate  possible  defects  in  doctrine,  performance  measures,  or  test  conditions. 

The  first  defect  you  would  look  for  is  an  expert's  failure  to  do  a  performance 
measure  as  required  by  the  test.  If  both  scorers  agree  that  an  expert  failed  a 
performance  measure  and  the  expert  insists  he  is  right,  the  test  might  conflict  with 
doctrine.  You  would  revise  the  performance  measure  or  test  conditions  so  that  the 
examinee  and  scorers  agree  that  the  test  conforms  with  doctrine. 

The  second  defect  you  would  look  for  is  disagreement  between  the  scorers  on  a 
performance  measure.  If  scorers  disagree,  your  test  might  have  an  unclear  or  unobserv¬ 
able  performance  measure.  You  would  need  to  rewrite  the  performance  measure  or 
change  the  test  conditions. 

The  third  defect  you  would  look  for  is  unacceptability.  You  would  have  the 
experts  answer  the  hands-on  test  questionnaire  to  get  their  opinions  about  the  relevance 
of  the  task  and  the  fairness  of  the  measurement. 
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Figure  A-ll.  Test  item  scores  for  exercise  on  validation  phase  of 
paper -and -pencil  test. 
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Practice  Situation.  During  the  expert  tryout  of  a  hands-on  test,  both  scorers  gave 
the  expert  a  NO-GO.  The  expert  who  was  tested  gave  these  answers  on  the  hands-on  test 
questionnaire  (Fig.  9-V): 


1.  Is  the  task  covered  by  the  test  a 
job  requirement? 


Yes  No 

X 


2.  Does  the  test  provide  a  fair  measure  _ X 

of  job  requirements? 


Your  Task.  Look  over  the  questionnaire  responses  (above)  and  the  completed 
score  sheets  in  Figure  A- 12.  Decide  whether  this  test  has  any  likely  defects.  If  it  does, 
also  answer  question  9. 

8.  Does  this  test  have  any  likely  defects?  Yes  No 

9.  If  so,  what  is  the  defect? 

a.  Conflict  with  doctrine?  Yes  No 

b.  Unclear  or  unobservable  performance  measure?  Yes  No 

c.  Unacceptable  to  expert?  Yes  No 


Scorer  1 

Task: 

:  Prepare  an  M72A2  LAW  for  firing;  engage  targets. 

Performance  Measures 

Pass 

Fail 

1. 

Removes  sling  assembly  and  extends  the  LAW 
until  locked  in  position. 

X 

2. 

Places  LAW  on  shoulder  with  front  end  of  LAW 
toward  the  target. 

X 

3. 

Checks  back  blast  area  before  arming  LAW. 

X 

— 

4. 

Arms  LAW  (by  pulling  safety  handle  out)  and 
simulates  firing  the  weapon  (by  pressing  the 
trigger  until  it  clicks)  with  weapon  on  shoulder. 

X 

5. 

Completes  all  performance  measures  in  30  seconds. 

X 

- - 

If  all  performance  measures  are  checked  pass,  mark 
GO.  If  any  performance  measure  is  checked  fail, 
mark  NO-GO. 

GO 

CD 

NO-GO 

m 

Scorer  2 

Task: 

Prepare  an  M72A2  LAW  for  firing;  engage  targets. 

Performance  Measures 

Pass 

Fail 

1. 

Removes  sling  assembly  and  extends  the  LAW 
until  locked  in  position. 

X 

2. 

Places  LAW  on  shoulder  with  front  end  of  LAW 
toward  the  target. 

X 

3. 

Checks  back  blast  area  before  arming  LAW. 

-  —  - 

X 

4. 

Arms  LAW  (by  pulling  safety  handle  out)  and 
simulates  firing  the  weapon  (by  pressing  the 
trigger  until  it  clicks)  with  weapon  on  shoulder. 

X 

5. 

Completes  all  performance  measures  in  30  seconds. 

X 

— - 

If  all  performance  measures  are  checked  pass,  mark 
GO.  If  any  performance  measure  is  checked  fail, 
mark  NO-GO. 

GO 

□ 

NO-GO 

m 

Figure  A- 12.  Experts'  score  sheets  for  hands-on  test  expert  tryout. 
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Hands-on  T est:  Validation 


i 


► 


h- 


Finally,  in  this  last  exercise,  you  will  practice  handling  validation  data  from 
hands-on  tests.  Remember  that  you  are  trying  to  see  if  experts  agree  when  they  score 
students  on  the  test.  You  will  use  five  students  and  four  scorers. 

Practice  Situation.  A  hands-on  test  has  just  been  scored  by  four  scorers.  Here 
are  the  results  for  each  of  the  five  students  who  took  it: 


Scorer 


Student 


12  3  4 

1  P  P  P  F 

2  P  P  P  P 

3  F  F  F  F 

4  F  F  F  F 

5  P  P  P  P 


P  =  Pass 
F  =  Fail 


Your  Task.  Compute  the  scorer  agreement  for  the  test  results  above.  Use  the 
table  below  to  record  agreement  (A)  or  disagreement  (D)  for  each  pair  of  scorers  for  each 
student. 


Scorer  Pair 


Student 


1&2 

1&3 

1&4 

2*3 

2&4 

3&4 

Total 
Agree  (A) 

Total 

Disagree  (D) 

1 

2 

3 

4 

5 

_ 

1 
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percent. 


10.  The  scorer  agreement  for  this  test  is _ 

Compute  scorer  agreement  for  these  test  results: 


Scorer 


1 

2 

Student  3 

4 

5 


12  3  4 
F  F  F  F 
P  F  P  F 
P  P  P  P 
P  P  P  P 
F  P  P  F 


11.  The  scorer  agreement  for  this  test  is 
Evaluation  Implementation 


percent. 


12.  When  you  are  using  two  forms  of  the  same  test  in  your  course,  the  best 
practice  is  to: 

a.  Use  form  A  as  a  pretest,  and  form  B  as  a  mastery  test. 

b.  Use  form  A  both  as  a  pretest  and  as  a  mastery  test. 

c.  Switch  back  and  forth  between  A  and  B  for  pretest  and  mastery  test. 

d.  Use  form  A  as  a  mastery  test  only. 

13.  Which  of  the  following  is  a  good  rule  to  follow  in  revising  the  answer  keys  to 
your  test? 


a.  Do  not  revise  any  key  for  the  first  3  months  of  the  course. 

b.  Do  not  revise  the  key  without  the  agreement  of  at  least  one  other 

instructor. 

c.  Do  not  revise  the  key  at  all. 

d.  Do  not  revise  the  key  unless  80  percent  or  more  students  fail  to  master 
the  objectives. 

14.  What  is  the  best  practice  when  scoring  tests? 


a.  Grade  "easy." 

b.  Grade  "hard." 

c.  Grade  the  same  way  for  all  students. 

d.  Grade  according  to  student's  individual  characteristics. 
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Answers  (Chapter  9) 

1.  Items  3  and  5  are  questionable.  Any  item  that  an  expert  fails  is 
questionable;  therefore,  there  is  no  doubt  about  item  5  since  Stengal  failed  it.  You  should 
also  follow  up  on  any  item  that  an  expert  is  unsure  about,  which  in  this  case  means  item  3, 
based  on  Alston's  response  on  the  questionnaire  (Table  A-l). 

2.  You  should  interview  Alston  (about  items  3  and  5)  and  Stengal  (about  item 
5).  Even  if  an  expert  does  not  indicate  a  specific  weakness  in  an  item,  interview  him  if  he 
selects  answers  B  or  C  to  item  5. 


3. 

Be  prepared  to  do  any  of  the  following  (listed  in  no  particular  order)  to 

item  3. 

a. 

Rewrite  situation. 

b. 

Rewrite  stem. 

c. 

Rewrite  alternatives. 

d. 

Add  likely  incorrect  alternative(s). 

Be  prepared  to  do  any  of  the  following,  listed  in  order  of  probability,  to 

item  5. 

a. 

Check  that  the  alternative  Stengal  chose  is  incorrect. 

b. 

Rewrite  stem. 

c. 

Rewrite  situation. 

d. 

Rewrite  alternative. 

e. 

Check  that  correct  alternative  is  correct. 

4. 

24. 

(6  x  11) -(7x6). 

5. 

90. 

(9  x  12)  -  (6  x  3). 

6. 

1 

v*> 

o 

(7  x  5)- (13  x  5). 

7. 

c. 

Only  item  3  must  be  revised. 

8. 

Yes. 

Defects  are  indicated  by  the  scorer  disagreement  on  performance 

measures  3  and  4. 

9.  The  most  likely  defect  is  unclear  or  unobservable  performance  measures. 
There  may  also  be  a  conflict  with  doctrine.  But  you  need  to  revise  the  test  so  that 
scorers  can  agree  on  what  happened  before  you  can  determine  whether  what  happened 
conforms  to  doctrine. 
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10.  The  scorer  agreement  for  this  test  is  90  percent  (27  *  30),  which  is  an 
acceptable  level.  Your  table  should  look  like  this: 


Scorer  Pairs 


1&2 

1&3 

lic4 

23c3 

2&4 

3&4 

Total 
Agree  (A) 

Total 

Disagree 

1 

A 

A 

D 

A 

D 

D 

3 

3 

2 

A 

A 

A 

A 

A 

A 

6 

0 

Student 

3 

A 

A 

A 

A 

A 

A 

6 

0 

4 

A 

A 

A 

A 

A 

A 

6 

0 

5 

A 

A 

A 

A 

A 

A 

6 

0 

27  3 

11.  73-1/3  percent.  (Unacceptable) 

12.  c. 

13.  b. 

14.  c.  Save  the  individualization  for  the  instruction  phase  of  your  system. 
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TEST  FORM  A 


1.  Which  one  of  these  most  distinguishes  an  individualized  system  of  instruction  from 
nonindividualized  systems? 

a.  It  uses  audiovisual  media  to  instruct  students. 

b.  It  treats  students  differently  according  to  performance. 

c.  It  is  based  on  training. 

d.  It  uses  pass/fail  grading. 

2.  Classify  the  following  hypothetical  instruction  system  as  individualized  or  nonin¬ 
dividualized:  "Students  enter  the  course  at  various  times  of  the  week.  A  student  views  a 
series  of  videotaped  presentations  on  first  aid.  When  a  student  has  seen  each  videotape 
presentation  twice,  he  or  she  is  graduated." 

a.  Individualized. 

b.  Not  individualized. 

3.  Which  one  of  the  following  is  a  basic  foundation  of  individualized  instruction? 

a.  Differences  among  students  will  cause  them  to  learn  at  different  rates. 

b.  The  lecture  is  the  most  important  instructional  technique  of  any  course. 

c.  A  wide  variety  of  media  are  essential  to  training  any  group  of  students. 

d.  All  students  should  reach  the  same  level  of  achievement  in  the  same  amount  of 
time. 

Classify  the  instruction  systems  described  in  items  *  through  8  by  checking  other 
"Fixed"  or  "Variable"  for  each  factor  listed. 

4.  All  students  attend  the  daily  lecture  on  "Examples  of  Leadership."  The  class  then 
breaks  into  conference  groups  to  discuss  the  lecture.  After  lunch,  students  complete 
written  exercises  in  a  workbook  on  leadership  and  management  techniques.  Students  who 
have  completed  all  exercises  in  the  workbook  take  a  test  on  the  following  day.  Each 
objective  covered  on  the  test  is  scored  pass/fail.  The  student  who  pass  the  test  graduate; 
those  who  do  not  complete  tape/slide  exercises  on  the  objectives  they  have  failed  and 
then  retake  the  test. 

Fixed  Variable 


Content 


Time 


Method 


Achievement  level 


5.  Each  student  entering  the  course  is  given  from  7  to  15  workbooks  to  study,  depending 
on  the  duties  involved  in  his  or  her  job  assignment.  Each  workbook  covers  a  different  set 
of  tasks.  As  soon  as  the  student  has  passed  a  test  for  one  workbook,  he  or  she  begins  the 
next  and  continues  in  this  way  until  all  assigned  workbooks  have  been  completed.  Any 
student  who  fails  a  workbook  test  studies  the  workbook  and  retakes  the  test  until  he  or 
she  has  attained  the  passing  score. 
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Fixed 


Variable 


Content 


Time 


Method 


6.  All  students  study  the  same  text  using  the  questions  in  a  study  guide  to  check  their 
progress.  As  soon  as  they  finish  a  chapter  in  the  study  guide,  they  take  a  test  for  that 
chapter.  Students  who  pass  the  test  advance  to  the  next  chapter  and,  when  all  chapters 
have  been  completed,  they  graduate  from  the  course.  Any  student  who  fails  a  chapter 
test  must  restudy  the  chapter  and  retake  the  test  until  he  or  she  passes  it. 

Fixed  Variable 


Content 


Time 


Method 


Achievement  level 


7.  There  are  eight  modules  in  the  course  and,  for  each  module,  there  are  two  sets  of 
worksheets.  As  soon  as  a  student  has  completed  all  worksheets  in  Set  1  for  a  module,  he 
or  she  is  tested.  A  student  who  passes  the  test  advances  to  the  next  module,  but  a  student 
who  fails  the  module  test  completes  all  worksheets  in  Set  2  before  being  tested  again.  A 
student  is  graduated  from  the  course  as  soon  as  he  or  she  has  passed  all  eight  modules. 

Fixed  Variable 


Content  _  _ 

Time  _  _ 

Method  _  _ 

Achievement  level  _  _ 

8.  The  course  consists  of  27  hands-on  performance  test  stations.  To  prepare  for  each 
performance  test,  each  student  practices  with  a  peer  or  buddy.  As  soon  as  a  student  has 
passed  a  performance  test,  he  or  she  begins  to  train  for  the  next  test.  When  all  tests  have 
been  passed,  the  student  is  graduated  from  the  course. 

Fixed  Variable 


Content 

Time 

Method 

Achievement  level 
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9.  Which  one  of  the  following  is  the  best  definition  of  an  audiovisual  medium? 

a.  A  system  of  individualized  instruction. 

b.  A  type  of  programmed  instruction. 

c.  A  way  to  present  instruction. 

d.  An  alternative  to  videotape. 

10.  Which  of  the  following  can  be  allowed  to  vary  in  a  single  individualized  system? 

a.  Content  and  time  only. 

b.  Time  and  method  only. 

c.  Content,  time,  and  method  only. 

d.  Content,  time,  method,  and  achievement  level. 

11.  There  are  at  least  five  functions  that  an  instructor  should  perform  in  an  individ¬ 
ualized  course.  One  of  them  is  to  give  feedback.  Name  two  others: 


a. 


b. 


12.  In  an  individualized  course,  what  should  an  instructor  do  first  if  a  student  does  not 
understand  part  of  a  lesson? 

a.  Assign  additional  study. 

b.  Switch  to  an  alternate  medium. 

c.  Retest  the  student. 

d.  Determine  why  the  student  does  not  understand  it. 

13.  Which  one  of  the  following  would  provide  the  most  effective  feedback  to  a  student? 

a.  The  correct  answers  to  a  set  of  practice  questions. 

b.  The  percentage  score  on  a  practice  test. 

c.  A  pep  talk  for  making  a  good  effort  on  a  set  of  practice  questions. 

d.  A  chance  to  do  the  exercise  over  again. 

14.  One  of  your  students  is  reading  the  correct  answers  to  some  study  questions.  This  is 
an  example  of: 


a.  Evaluation. 

b.  Student  reponse. 

c.  Feedback. 

d.  Presentation. 


15.  One  of  your  students  is  practicing  converting  simulated  fire  request  information  to 
firing  data.  This  is  an  example  of: 


a.  Feedback. 

b.  Student  response. 

c.  Presentation. 

d.  Evaluation. 


B-3 


J 


16.  Student  response  must  always  be  followed  directly  by  which  one  of  these  parts  of  an 
instruction  system? 

a.  Evaluation. 

b.  Feedback. 

c.  Presentation. 

d.  None  of  these. 

17.  Evaluation  is  particularly  important  in  individualized  instruction.  Select  the  main 
role  that  tests  play  in  an  individualized  system  that  they  do  not  play  in  a  lock-step 
system. 

a.  Tests  determine  what  the  student  knows. 

b.  Tests  are  based  on  objectives. 

c.  Tests  are  graded  pass/ fail. 

d.  Tests  control  the  student's  progress. 

18.  Which  one  of  the  following  is  the  basic  pattern  of  instruction? 

a.  Presentation,  evaluation,  student  response,  feedback. 

b.  Student  response,  feedback,  presentation,  evaluation. 

■  c.  Presentation,  student  reponse,  feedback,  evaluation. 

d.  Student  response,  evaluation,  presentation,  feedback. 

19.  Two  parts  of  the  instruction  are  often  repeated  several  times  within  the  basic 
pattern.  Which  pair  are  they? 

a.  Evaluation  and  student  response. 

b.  Student  response  and  feedback. 

c.  Presentation  and  feedback. 

d.  Student  response  and  presentation. 

20.  At  some  point  in  your  system,  you  might  use  instructor  decision  as  a  way  to 
determine  what  the  student  should  do  next.  Which  one  of  the  following  is  a  disadvantage 
of  including  instructor  decision  in  your  system? 

i 

a.  The  instructor  is  a  poor  judge  of  what  a  student  needs  to  learn. 

b.  The  instructor  does  not  have  enough  information  to  decide  what  the  student 
should  do  next. 

c.  The  instructor  should  have  a  hands-off  attitude  in  individualized  instruction 
systems. 

d.  The  instructor  might  be  swamped  by  too  many  decisions  to  make  in  a  short  time. 

21.  A  student  in  your  class  has  just  demonstrated  mastery  of  part  of  your  instruction. 
Which  component  of  instruction  should  come  next? 

a.  Presentation. 

b.  Evaluation. 

c.  Student  response. 

d.  Restudy  of  the  previous  lesson. 
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22.  One  of  your  students  has  viewed  all  the  presentations  for  a  lesson,  completed  all  the 
student  activities,  and  has  been  evaluated.  What  should  the  next  component  of  instruction 
be? 

a.  Evaluation. 

b.  Presentation. 

c.  Student  response. 

d.  It  depends  on  the  results  of  the  evaluation. 

23.  Situation:  You  are  about  to  use  the  IQI  in  converting  an  existing  course  into  an 
individualized  course.  You  should  first  make  sure  that: 

a.  Your  presentations  are  consistent  with  your  objectives. 

b.  Your  objectives  are  complete  and  correctly  stated. 

c.  Your  test  items  are  adequate. 

.  d.  Your  practice  items  are  adequate. 

24.  To  make  sure  your  tests  are  consistent  with  your  objectives,  you  could  use  the  IQI  to 
classify  them  first.  One  purpose  of  checking  for  consistency  between  test  items  and 
objectives  is  to  make  sure  that: 

a.  Tests  are  scored  accurately. 

b.  Objectives  are  clearly  stated. 

c.  The  content  being  tested  is  the  same  as  the  content  being  taught. 

d.  The  content  of  the  course  is  job-oriented. 

25.  Which  one  of  the  following  is  an  IQI  rule  for  constructing  good  test  items? 

a.  Items  should  not  give  away  the  answer  to  other  items. 

b.  Items  should  be  very  brief. 

c.  Items  should  not  use  "scenarios." 

d.  Items  should  ask  for  recall  of  information,  not  application. 

26.  You  are  developing  an  individualized  course.  You  have  determined  that  students  will 
hand  their  exercises  to  a  student  grader,  not  to  the  instructor.  Which  of  the  following  is 
the  best  way  to  get  the  student  to  do  this? 

a.  Have  the  instructor  issue  "daily  reminders." 

b.  Include  this  information  in  the  student's  instruction. 

c.  Print  this  information  on  the  student  progress  card  that  the  instructor  has  at  the 
control  desk. 

d.  Leave  this  decision  up  to  the  classroom  instructor. 

27.  There  is  a  good  reason  to  provide  space  for  start  time  and  estimated  completion  time 
right  on  the  student  progress  card.  What  is  that  reason? 

a.  It  will  help  the  instructor  know  whether  the  student  may  be  having  problems. 

b.  The  student  will  fail  the  lesson  if  he  completes  it  late. 

c.  The  instructor  always  should  test  at  the  end  of  the  estimated  completion  time. 

d.  This  information  determines  which  test  form  the  student  will  be  given. 
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28.  Which  of  these  is  the  best  way  to  inform  a  student  of  how  much  time  a  lesson  should 
take? 

a.  Post  a  list  of  times  on  the  bulletin  board. 

b.  Leave  this  to  the  classroom  instructor's  discretion. 

c.  Print  this  information  in  the  instructor's  guide. 

d.  Include  this  information  in  the  student's  printed  or  audiovisual  instruction. 

29.  In  the  system  shown  below,  what  will  a  student  do  after  completing  worksheets 
part  2? 

a.  Enter  next  module. 

b.  Take  the  module  test. 

c.  Be  "recycled." 

d.  It  is  the  instructor's  decision. 


30.  In  the  system  shown  below,  what  would  a  student  do  if  he  or  she  passed  the  test  on 
the  third  try? 

a.  Receive  more  instruction. 

b.  Receive  a  diagnosis  and  prescription. 

c.  Go  to  the  next  lesson  after  test  review. 

d.  It  is  the  instructor's/school  director's  decision. 


name _  LESSON 
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31.  In  the  system  shown  below,  how  many  lesson(s)  would  a  student  take  after  the 
diagnostic  test? 

a.  One. 

b.  Two. 

c.  Three. 

d.  It  depends  on  the  results. 
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32.  Which  one  of  the  following  should  you  do  first  in  the  process  of  developing  tests  for 
your  course? 


a.  Select  a  sampling  strategy. 

b.  Determine  the  criticality  of  tasks. 

c.  Define  the  performance  domain. 

d.  Sample  on  the  basis  of  frequency. 

33.  Why  is  it  necessary  to  sample  the  performance  domain? 

a.  Not  every  element  of  performance  can  be  represented  on  the  test. 

b.  Some  tasks  in  the  performance  domain  of  the  test  are  not  worth  testing. 

c.  The  performance  domain  is  defined  well  enough. 

d.  Tests  should  include  as  few  items  as  possible. 

34.  Which  of  the  following  test  forms  would  be  best  for  a  motor  skill? 

a.  Paper-and-pencil  test. 

b.  Hands-on  test. 

c.  There  is  no  basis  for  choosing. 

d.  Neither  is  acceptable. 


35.  Of  the  following  four  methods  for  sampling  the  performance  domain,  which  one 
requires  identifying  commonalities  and/or  dependencies  among  tasks? 


a.  Random. 

b.  Generality. 

c.  Frequency. 

d.  Criticality. 


36.  Which  one  of  the  following  is  one  advantage  of  paper-and-pencil  tests  (over  hands-on 
tests)? 


a.  They  have  lower  reading  demands. 

b.  They  are  more  relevant  to  job  performance. 

c.  They  can  cover  more  tasks. 

d.  They  represent  problem  solving  better. 

37.  Which  one  of  the  following  is  most  characteristic  of  applying  a  rule  or  principle? 

a.  Recognizing  hidden  cues. 

b.  Using  strength  and  good  balance. 

c.  Remembering  a  step-by-step  sequence. 

d.  Figuring  out  which  rule  applies. 

38.  Free  response  (fill-in-the-blank)  test  items  can  be  used  only  when: 


a.  The  question  can  be  answered  briefly,  with  a  few  words  or  sentences. 

b.  You  are  testing  for  remembering  a  fact. 

c.  The  question  concerns  a  rule  or  principle. 

d.  The  question  can  be  answered  without  looking  at  references. 


39.  True-false  test  items  can  be  used  only  to  test: 


a.  Procedural  tasks. 

b.  Statements  that  are  always  true  or  false. 

c.  Statements  taken  from  textual  material. 

d.  Long  and  complicated  statements. 

40.  One  advantage  that  multiple-choice  test  items  have  over  free  response  items  is  that 
multiple-choice  items: 

a.  Are  less  subjective  to  score. 

b.  Are  easier  to  construct. 

c.  Can  cover  more  tasks  in  a  single  test. 

d.  Are  harder  for  the  student  to  guess  at. 

41.  Read  the  following  two  sets  of  performance  measures.  Then,  select  the  best  answer: 

a.  Both  sets  are  product- scored. 

b.  Both  sets  are  process-scored. 

c.  Only  set  1  is  product-scored. 

d.  Only  set  2  is  product-scored. 


SET  1: 


A.  Gave  signal  for  ASSEMBLE.  (At  position  of  attention, 
raised  arm  vertically  overhead,  palm  to  the  front,  anc 
waved  in  large  horizontal  circles.) 

B.  Gave  signal  for  LINE  FORMATION.  (At  position  of  atten¬ 
tion,  raised  both  arms  to  the  side  until  horizontal  with 
arms  and  hands  extended  and  palms  down.) 

C.  Gave  signal  for  ATTENTION.  (At  position  of  attention, 
extended  the  arm  sideways  above  horizontal,  palm  to  the 
front,  and  waved  arm  to  and  away  from  the  head  several 
times.) 


SET  2: 


A.  Stated  known  point  as  ’’From  Registration 


B.  Stated  observer-target  azimuth  as  "DIRECTION 


c. 

Stated  lateral 
"RIGHT/LEFT _ 

shift 

II 

from 

known 

point, 

if 

any, 

as 

D. 

Stated  range 
"ADD/DROP _ 

shift 

II 

• 

from 

known 

point, 

if 

any, 

as 

E. 

Stated  vertical 
"UP/DOWN _ 

shift 

tt 

• 

from 

known 

point, 

if 

any, 

as 
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42.  One  advantage  of  product-scoring  is  that: 


a.  You  do  not  have  to  test  as  many  students. 

b.  The  most  important  tasks  are  product-scored. 

c.  You  can  sometimes  test  more  than  one  student  at  a  time. 

d.  There  are  fewer  performance  measures. 

43.  In  which  case  should  you  indicate  to  the  scorer  whether  the  sequence  of  steps  will  be 
a  scoring  point  in  a  hands-on  test? 

a.  Only  when  sequence  will  be  scored. 

b.  Only  when  some  steps  must  be  in  order. 

c.  In  ail  hands-on  tests. 

d.  In  no  hands-on  tests. 

44.  Which  is  the  best  description  of  how  the  test  station  should  be  set  up  for  a  hands-on 
test? 

a.  A  different  way  for  every  other  student. 

b.  At  the  discretion  of  the  station  test  manager. 

c.  At  random. 

d.  The  same  way  every  time. 

45.  What  is  one  purpose  for  conducting  an  expert  tryout  of  a  paper-and-pencil  test? 

a.  To  find  test  items  that  disagree  with  doctrine. 

b.  To  measure  the  average  skill  level  of  job  experts. 

c.  To  see  if  the  experts  are  good  enough  to  serve  as  test  scorers. 

d.  To  find  test  items  that  are  easy  to  construct. 

46.  One  purpose  of  validation  of  paper-and-pencil  tests  is  to  see  if: 

a.  Experts  get  all  the  items  correct. 

b.  The  average  score  is  passing. 

c.  The  test  is  long  enough. 

d.  The  test  can  separate  better  performers  from  poorer  performers. 

47.  One  purpose  of  the  validation  of  a  hands-on  test  is  to  see  if  all  performance  measures 
are: 

a.  Passed  by  every  student. 

b.  Equally  difficult. 

c.  Scored  the  same  way  by  all  scorers. 

d.  Performed  as  fast  as  possible. 

48.  You  have  found  that  your  students  score  an  average  of  12  points  higher  on  form  B  of 
your  lesson  test  than  on  form  A.  Which  of  the  following  methods  is  best  for  equalizing  the 
two  forms? 

a.  Move  items  from  one  test  to  the  other. 

b.  Allow  longer  time  for  the  harder  test. 

c.  Cut  out  some  items  from  the  harder  test. 

d.  None  of  these  is  acceptable. 
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49.  Below  are  some  data  from  the  validation  of  a  paper-and-pencil  test.  Compute  the 
agreement  index. 

Masters  Nonmasters 
Pass  6  6 

Fail  6  11 

Formula:  (A  x  D)  -  <B  x  C)  =  Agreement  index 
Agreement  index  = _ . 


50.  Below  are  some  data  from  the  validation  of  a  hands-on  test.  Compute  the  scorer 
agreement. 

Scorer  Pair 


1&2 

1&3 

1&4 

2&3 

2&4 

3&4 

Total 

Agree(A) 

Total 

Disagree(D) 

1 

A 

A 

A 

A 

A 

A 

6 

0 

2 

A 

D 

D 

D 

D 

A 

2 

4 

Student  3 

D 

A 

A 

D 

D 

A 

3 

3 

4 

A 

A 

A 

A 

A 

A 

6 

0 

5 

A 

A 

D 

A 

D 

D 

3 

3 

Scorer  agreement  = 

_percent. 
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Answer  Key  (Test  Form  A) 


1. 

b. 

17. 

d. 

2. 

b. 

18. 

c. 

3. 

a. 

19. 

b. 

4. 

Fixed  content. 

20. 

d. 

Variable  time  (to  study). 

21. 

a. 

Variable  method. 

22. 

d. 

Fixed  achievement  level. 

23. 

b. 

5. 

Variable  content. 

24. 

c. 

Fixed  time. 

25. 

a. 

Fixed  method. 

26. 

b. 

6. 

Fixed  content. 

27. 

a. 

Variable  time. 

28. 

d. 

Fixed  method. 

29. 

b. 

Fixed  achievement  level. 

30. 

c. 

7. 

Fixed  content. 

31. 

d. 

Variable  time. 

32. 

c. 

Fixed  method. 

33. 

a. 

Fixed  achievement  level 

34. 

b. 

8. 

Fixed  content. 

35. 

b. 

Variable  time. 

36. 

c. 

Fixed  method. 

37. 

d. 

Fixed  achievement  level. 

38. 

a. 

9. 

c. 

39. 

b. 

10. 

d. 

40. 

a. 

11. 

Any  two  of  the  following: 

41. 

b. 

(a)  Give  encouragement  (motivation). 

42. 

c. 

(b)  Diagnose  problems. 

43. 

c. 

(c)  Counsel. 

44. 

d. 

(d)  Give  direct  instruction. 

45. 

a. 

12. 

d. 

46. 

d. 

13. 

a. 

47. 

c. 

14. 

c. 

48. 

a. 

15. 

b. 

49. 

30. 

16. 

b. 

50. 

66-2/3  percent  (or  67%), 
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TEST  FORM  B 


1.  Which  one  of  the  following  is  the  best  description  of  individualized  instruction? 

a.  Audiovisual  media  used  to  teach  students  individually. 

b.  Systematic  use  of  programmed  texts. 

c.  Instruction  based  on  learning  objectives  and  job  performance. 

d.  Treating  each  student  differently  on  a  systematic  basis. 

2.  In  most  individualized  instruction  systems,  how  are  instructional  decisions  made 
about  individual  students  during  the  course? 

a.  On  the  basis  of  aptitude  scores  at  the  beginning  of  the  course. 

b.  On  the  basis  of  reading  ability  tests. 

c.  On  the  basis  of  performance  during  the  course. 

d.  On  the  basis  of  instructors'  ratings  of  individuals?  motivation. 

3.  Which  one  of  the  following  is  mentioned  in  your  text  as  a  disadvantage  of  live 
presentations  over  audiovisual  presentations? 

a.  Live  presentations  are  usually  based  on  incorrect  information. 

b.  Live  presentations  cannot  be  illustrated. 

c.  Live  presentations  cannot  cover  an  entire  subject. 

d.  Live  presentations  are  not  easily  looked  at  again. 

Classify  the  instruction  systems  described  in  items  *  throqgh  8  by  checking  either  "Fixed" 
or  "Variable"  for  each  factor  listed. 

4.  Students  select  from  two  to  six  areas  for  individual  study.  For  each  area  chosen,  the 
students  read  a  text  book  of  75-100  pages.  After  reading  each  textbook,  they  take  a 
multiple-choice  test.  The  students  who  fail  the  test  restudy  the  textbook  until  they  can 
pass  the  test.  Then,  they  study  the  next  area. 

Fixed  Variable 


Content 

Time 

Method 


5.  Students  each  receive  from  6  to  13  lesson  workbooks  to  study,  depending  on  which 
tasks  they  will  be  performing  in  the  field.  When  they  have  mastered  one  lesson,  they 
begin  the  next.  To  demonstrate  mastery  of  a  lesson,  students  must  score  90  percent  on  a 
lesson  test.  Students  who  fail  a  lesson  test  must  restudy  the  workbook  and  retake  the  test 
until  they  pass. 


Fixed  Variable 


Content 

Time 


Method 


J 


I 


6.  Each  student  receives  a  cooy  of  the  course  study  guide  and  textbook  at  the  start  of 
the  course.  The  student  uses  the  study  guide  to  prepare  for  a  series  of  lesson  or  chapter 
tests.  When  the  student  has  passed  the  test  for  one  lesson,  he  or  she  advances  to  the  next. 
A  student  who  fails  a  lesson  test  restudies  the  study  guide  and  the  textbook  and  retakes 
the  lesson  test  until  passing  it. 

Fixed  Variable 


Content 

Time 

Method 

Achievement  level 


7.  The  course  materials  consist  of  seven  lesson  tests  and  three  sets  of  worksheets  for 
each  lesson.  Each  student  begins  each  lesson  by  completing  the  first  set  of  worksheets  for 
that  lesson.  Then,  the  student  is  tested.  If  he  or  she  passes  the  lesson  test,  the  student 
begins  the  second  lesson;  if  not,  he  or  she  completes  the  second  set  of  worksheets;  and,  if 
he  or  she  fails  the  lesson  test  again,  the  third  set.  The  student  must  pass  each  lesson  to 
complete  the  course. 


Fixed  Variable 


Content 

Time 

Method 

Achievement  level 


8.  To  complete  the  course,  students  must  pass  each  of  16  hands-on  performance  tests. 
Students  practice  for  the  performance  tests  with  a  peer,  another  student  who  is  at  the 
same  point  in  the  course.  As  soon  as  a  student  has  passed  a  performance  test,  he  or  she 
begins  preparing  for  another  one. 


Fixed  Variable 


Content 

Time 

Method 

Achievement  level 
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9.  Which  one  of  the  following  is  the  best  description  of  programmed  instruction? 

a.  A  medium  of  communication. 

b.  An  individualized  instruction  system. 

c.  An  alternative  to  teaching  machines. 

d.  A  method  of  conducting  instruction. 

10.  Which  of  the  following  audiovisual  media  is  the  easiest  to  revise  and  update? 

a.  Motion  pictures. 

b.  "Cue/see"  filmstrip. 

c.  Videotape. 

d.  All  three  are  equally  easy  to  revise. 

11.  Which  one  of  the  following  instructor  functions  is  the  most  important  in  an 
individualized  course? 

a.  Grading  multiple-choice  tests. 

b.  Diagnosing  students'  problems  with  the  instruction. 

c.  Cataloging  instructional  materials  and  audiovisual  devices. 

d.  Keeping  records  of  students'  rates  of  progress. 


12.  According  to  the  text,  an  instructor  should  give  feedback: 

a.  In  written  form. 

b.  Only  after  evaluation. 

c.  During  a  test. 

d.  Soon  after  the  student's  response. 

13.  Which  one  of  the  following  is  the  basic  pattern  of  instruction. 

a.  Presentation,  student  response,  feedback,  evaluation. 

b.  Presentation,  evaluation,  student  response,  feedback. 

c.  Student  response,  feedback,  presentation,  evaluation. 

d.  Student  response,  evaluation,  presentation,  feedback. 

14.  One  of  your  students  is  completing  a  set  of  practice  questions  in  a  study  guide.  This 
is  an  example  of: 


a.  Evaluation. 

b.  Presentation. 

c.  Feedback. 

d.  Student  response. 


15.  One  of  your  students  is  watching  a  videotaped  demonstration  of  how  to  conduct  an 
inspection.  This  is  an  example  of: 


l 


a.  Presentation. 

b.  Student  response. 

c.  Evaluation. 

d.  Feedback. 
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16.  One  of  your  students  is  reading  how  a  computer  works.  This  is  an  example  of: 


a.  Student  respose. 

b.  Presentation. 

c.  Evaluation. 

d.  Feedback. 


17.  One  of  your  students  is  listening  to  an  instructor's  comments  about  her  analysis  of 
British  defense  policy.  This  is  an  example  of: 


a. 

Student  response. 

b. 

Evaluation. 

c. 

Feedback. 

d. 

Presentation. 

18.  One 

of  your  students  is  practicing  night  compass  reading. 

This  is  an  example  of: 

a. 

Student  response. 

b. 

Evaluation. 

c. 

Presentation. 

d. 

Feedback. 

19.  One 

of  your  students  is  looking  at  the  "school  solution" 

'  to  a  problem  in  defensive 

operations.  This  is  an  example  of: 

a. 

Student  response. 

b. 

Evaluation. 

c. 

Feedback. 

d. 

Presentation. 

20.  According  to  the  text,  evaluation  is  particularly  important  in  an  individualized 
system,  because  in  an  individualized  course: 

a.  Training  objectives  are  more  difficult  to  achieve. 

b.  Evaluation  determines  what  a  student  will  do  next. 

c.  Audiovisual  media  are  often  used. 

d.  Presentations  are  not  usually  made  by  the  instructor. 

21.  According  to  the  instructional  quality  inventory  0QI)  described  in  the  text,  which  one 
of  the  following  rules  should  you  follow  for  the  presentation  part  of  your  lessons? 

a.  Use  printed  lessons  rather  than  audiovisual  lessons. 

b.  Never  present  two  important  points  in  the  same  lesson. 

c.  Separate  important  points  from  the  rest  of  the  lesson  material. 

d.  Repeat  important  points  at  least  twice. 

22.  Which  one  of  the  following  is  an  IQI  rule  for  developing  good  test  items? 

a.  Items  should  always  be  short. 

b.  Directions  should  refer  to  previous  test  items. 

c.  Items  should  be  arranged  in  order  from  easy  to  hard. 

d.  Directions  should  not  be  ambiguous. 


23.  According  to  the  IQI,  you  should  classify  your  objectives,  tests,  and  presentations  to 
make  them  consistent.  One  reason  to  make  these  three  elements  consistent  is: 

a.  If  these  elements  are  not  consistent,  the  course  will  be  too  short. 

b.  When  these  elements  are  not  consistent,  students  will  tend  to  study  the 
objectives  only  and  not  the  presentations. 

c.  If  you  teach  a  student  one  thing  and  test  him  on  another,  you  may  get  poor 
results. 

d.  Making  sure  these  elements  are  consistent  greatly  reduces  course  development 
time. 

2k.  In  the  individualized  course  you  are  planning,  you  have  decided  that  a  student  may 
"challenge"  the  lesson  test  at  any  time.  Which  one  of  the  following  is  the  best  way  to  let 
the  student  know  this? 

a.  Remind  the  instructors  daily  to  remind  all  students  of  this  rule. 

b.  Leave  this  matter  up  to  the  classroom  instructor. 

c.  Include  this  information  in  each  lesson  the  student  studies. 

d.  Print  this  rule  in  the  instructor's  guide. 

25.  In  an  individualized  instruction  system,  group  activities: 

a.  Can  be  scheduled  for  a  regular  time  in  advance. 

b.  Are  not  appropriate. 

c.  Are  difficult  to  schedule  and  to  manage  effectively. 

d.  Should  not  be  necessary. 

26.  Suppose  you  are  developing  a  course  in  which  the  passing  score  for  lessons  ]  through  k 
is  90  percent  and  the  passing  score  for  lessons  5  and  6  is  85  percent.  Which  of  the 
following  would  be  the  best  way  to  communicate  this  to  the  instructor? 

a.  Print  this  information  in  the  instructor's  guide. 

b.  Post  a  notice  about  this  in  the  instructors'  office. 

c.  Print  the  passing  score  on  the  student  progress  card  for  each  lesson. 

d.  Leave  this  matter  up  to  the  classroom  instructor. 
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28.  In  the  system  shown  below,  what  would  a  student  do  if  he  or  she  failed  the  test  for 
the  third  time? 


a.  It  is  the  instructor's/school  director's  decision. 

b.  Be  retested  immediately. 

c.  Receive  a  diagnosis  and  prescription. 

d.  Go  to  the  next  lesson  after  test  review. 


I 


29.  In  the  system  shown  below,  what  would  a  student  do  next  after  completing  all 
numbered  lessons? 

a.  Be  "recycled." 

b.  Take  the  mastery  test. 

c.  It  is  the  instructor's  decision. 

d.  Take  the  diagnostic  test. 
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30.  In  planning  a  test,  which  one  of  the  following  should  you  do  first? 

a.  Determine  what  you  are  going  to  measure. 

b.  Detemine  whether  to  use  a  paper-and-pencil  test. 

c.  Determine  how  long  the  test  should  be. 

d.  Determine  how  you  will  sample  the  domain.. 

31.  Which  one  of  the  following  is  a  good  reason  to  include  frequently  performed  tasks  in  a 
test? 

a.  They  are  obviously  important  to  the  job. 

b.  They  are  easier  to  perform. 

c.  They  usually  have  high  error  rates. 

d.  Many  performers  cannot  perform  them. 

32.  Which  one  of  the  following  methods  for  sampling  the  performance  domain  does  not 
depend  on  knowing  some  characteristic  of  the  performance  elements  you  select? 

a.  Criticality. 

b.  Frequency. 

c.  Generality. 

d.  Random. 

33.  A  particular  task  to  be  tested  includes  installing  a  grounding  rod  for  a  telephone  set, 
according  to  complete  written  instructions.  This  performance  element  requires  primarily: 

a.  Skilled  motor  coordination. 

b.  Application  of  a  general  principle. 

c.  Decision-making  skill. 

d.  Direct  application  of  a  procedure. 

34.  A  task  to  be  tested  includes  selecting  the  best  kind  of  solder  to  join  two  different 
types  of  metals.  This  element  of  performance  is  an  example  of: 

a.  Skilled  motor  performance. 

b.  Application  of  rules  or  principles. 

c.  Physical  endurance. 

d.  Troubleshooting  skill. 

35.  A  task  to  be  tested  includes  adjusting  the  controls  of  a  gunsight  to  keep  its  cross¬ 
hairs  on  a  moving  target.  This  element  of  performance  is  an  example  of: 

a.  Troubleshooting  skill. 

b.  Perceptual/motor  skill. 

c.  Application  of  general  principles. 

d.  Problem-solving  skill. 

36.  A  good  way  to  tell  If  a  task  includes  motor  and  perceptual  skill  Is  to  find  out  if  it: 

a.  Can  be  done  without  practice. 

b.  Is  done  frequently. 

c.  Requires  special  equipment. 

d.  Is  done  by  the  more  experienced  personnel. 
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37.  A  disadvantage  of  hands-on  tests,  compared  to  paper-and-pencil  tests,  is  that  hands- 
on  tests: 


a.  Are  more  likely  to  ask  for  irrelevant  information. 

b.  Cost  more  to  administer. 

c.  Depend  more  on  reading  skill. 

d.  Must  be  based  on  learning  objectives. 

38.  Which  one  of  the  following  is  an  Important  reason  for  writing  a  "pool"  of  items  for 
any  paper-and-pencil  test? 

a.  Students  should  be  given  a  choice  of  which  items  they  will  answer. 

b.  Some  of  the  items  will  probably  be  discarded  as  a  result  of  test  tryouts. 

c.  Each  knowledge  element  should  be  tested  by  several  different  types  of  items. 

d.  In  a  training  course,  each  test  item  should  only  be  used  for  a  few  weeks  at  most. 

39.  Which  one  of  the  following  types  of  test  items  requires  the  student  to  recall  the  right 
answer  from  memory,  not  just  recognize  it? 

a.  True/false. 

b.  Free  response. 

c.  Multiple-choice. 

d.  Matching. 

40.  One  important  reason  not  to  use  true/false  test  items  is  that: 

a.  True/ false  items  are  too  difficult  for  students  to  answer. 

b.  True/false  items  can  only  cover  rules  and  principles. 

c.  Most  false  statements  are  hard  to  understand. 

d.  Few  statements  are  all  true  or  all  false. 

41.  An  important  rule  to  follow  in  constructing  matching  items  is  to: 

a.  Make  all  elements  on  the  lists  the  same  type. 

b.  Make  both  lists  very  long. 

c.  Use  only  negatively  stated  items. 

d.  Ask  only  about  nomenclature. 

42.  An  advantage  of  product-scoring  in  performance  tests  is  that  product-scoring: 

a.  Can  measure  motor  skills,  but  process-scoring  cannot. 

b.  Allows  a  wider  variety  of  learning  objectives  to  be  measured. 

c.  Does  not  require  the  use  of  scoring  aids. 

d.  Makes  It  easier  to  standardize  the  test  from  student  to  student. 

43.  An  Important  general  rule  to  follow  when  constructing  process-scored  measures  Is  to: 

a.  Put  only  a  single  action  into  each  measure. 

b.  Put  all  safety  performance  measures  at  the  beginning  of  the  test. 

c.  Avoid  the  use  of  scoring  aids. 

d.  Include  as  many  actions  as  possible  in  the  test. 
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44.  An  important  rule  to  follow  in  conducting  a  performance  test  is  to  : 

a.  Set  up  the  station  the  same  way  for  each  student  tested. 

b.  Allow  the  student  to  use  any  job  manuals  he  or  she  brings  to  the  test. 

c.  Vary  the  layout  of  the  station  to  maintain  test  security. 

d.  Conduct  the  test  outdoors  in  daylight  on  a  clear  day. 

45.  You  are  about  to  conduct  the  expert  tryout  of  a  paper-and- pencil  test.  Which  one  of 
the  following  persons  would  be  the  best  test  subject  for  the  tryout? 

a.  An  E-7  who  had  just  been  assigned  as  an  instructor. 

b.  The  honor  graduate  of  the  most  recent  graduating  class. 

c.  A  new  student  who  had  not  gone  through  the  course  yet. 

d.  An  E-4  who  was  processing  out  of  the  service. 

46.  You  are  conducting  the  validation  of  a  paper-and- pencil  test  with  masters  and 
nonmasters.  For  a  test  item  to  be  acceptable: 

a.  A  greater  proportion  of  nonmasters  must  pass  the  item. 

b.  A  greater  proportion  of  masters  must  pass  the  item. 

c.  The  same  proportion  of  masters  and  nonmasters  should  pass  the  item. 

d.  A  majority  of  both  masters  and  nonmasters  should  pass  the  item. 

47.  You  are  conducting  the  expert  tryout  of  a  hands-on  test.  Which  one  of  the  following 
is  the  most  important  question  you  are  trying  to  answer? 

a.  Have  the  experts  practiced  the  task? 

b.  Would  nonexperts  pass  each  measure? 

c.  Which  one  of  the  experts  is  the  most  proficient  performer? 

d.  Do  persons  scoring  the  performance  measures  agree? 

48.  You  are  selecting  persons  to  be  tested  in  the  validation  of  a  hands-on  test.  Which 
type  of  test  subject  should  you  select? 

a.  Persons  who  have  never  done  the  task. 

b.  Persons  who  know  how  to  do  the  task. 

c.  Persons  who  are  just  learning  the  task. 

d.  Persons  who  are  not  sure  if  they  can  do  the  task. 

49.  Below  are  some  data  from  the  validation  of  a  paper-and- pencil  test.  Compute  the 
agreement  index. 


Masters  Nonmasters 
Pass  9  5 

Fail  6  12 

Formula:  (A  x  D)  -  (B  x  C)  *  Agreement  index 
Agreement  Index  = _ 
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50.  Below  are  some  data  from  the  validation  of  a  hands-on  test.  Compute  the  scorer 
agreement. 


Scorer  Pair 


1&2 

1&3 

1&4 

2&3 

2&4 

3&4 

Total 

A*ree(A) 

Total 

Disgaree(D) 

1 

D 

A 

A 

D 

D 

A 

3 

3 

2 

A 

A 

A 

A 

A 

A 

6 

0 

Student  3 

D 

D 

A 

A 

D 

D 

2 

4 

4 

D 

A 

D 

D 

A 

A 

3 

3 

5 

A 

A 

A 

A 

A 

A 

6 

0 

Scorer  agreement  = 

percent. 

Answer  Key  (Test  Form  B) 


1. 

d. 

20, 

2. 

c. 

21, 

3. 

d. 

22, 

4. 

Variable  content. 

23, 

Variable  time. 

24, 

Fixed  method. 

25, 

5. 

Variable  content. 

26, 

Variable  time. 

27. 

Fixed  method. 

28, 

6. 

Fixed  content. 

29, 

Variable  time. 

30, 

Fixed  method. 

31, 

Fixed  achievement  level. 

32. 

7. 

Fixed  content. 

33, 

Variable  time. 

34. 

Fixed  method. 

35, 

Fixed  achievement  level. 

36, 

8. 

Fixed  content. 

37, 

Variable  time. 

38, 

Fixed  method. 

39, 

Fixed  achievement  level. 

40. 

9. 

d. 

41, 

10. 

c. 

42, 

11. 

b. 

43. 

12. 

d. 

44. 

13. 

a. 

45. 

14. 

d. 

46. 

15. 

a. 

47, 

16. 

b. 

48. 

17. 

c. 

49. 

18. 

a. 

50. 

19. 

c. 

b. 

c. 

d. 
c. 
c. 
a. 
c. 

c. 

a. 

b. 
a. 

a. 

d. 
d. 

b. 
b. 

a. 

b. 
b. 
b. 
d. 
a. 
d. 
a. 
a. 

a. 

b. 
d. 
b. 

78. 

66-2/3  percent  (or  67%). 
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APPENDIX  C 
COURSE  FORMS 


Note.  Reproduce  these  forms  as  is  or  revise  them  as  needed  for  use  in  your  course. 


Page 

Sample  Student  Progress  Card .  C-l 

Individualization  System  Checklist  .  C-2 

Expert  Tryout  Questionnaire  .  C-3 

Suggested  Actions  if  Test  is  Not  Acceptable  to  Experts  .  C-4 

Table  For  Recording  Scores  to  Validate  Paper-and-peneil  Tests  .  C-5 

Computing  the  Agreement  Index .  C-6 

Hands-on  Test  Questionnaire  .  C-7 
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SAMPLE  STUDENT  PROGRESS  CARD 


OVERVIEW 


Date/Time_ 

Instr. 


LESSON 


^  TEST  ^ 
(Optional) 


Teat  Review 

Diagnosis 

Prescription 

.  Date/Time _ 

llpstr. _ 


INDIVIDUALIZATION  SYSTEM  CHECKLIST 

1.  Have  you  provided  a  means  for  the  student  to  know  how  to  get  started? 

2.  Have  you  provided  a  means  for  the  student  to  know  what  to  do  next? 

3.  Have  you  provided  a  means  for  the  student  to  know  the  lesson  objectives? 

_4.  Have  you  decided  whether  the  students  can  skip  the  lesson  pretest? 

_5.  Have  you  decided  whether  the  students  can  "challenge"  the  lesson  test? 

_6.  Have  you  provided  a  means  for  the  student  to  know  if  she  or  he  is  ready 

for  the  lesson  test? 

_7.  Have  you  provided  a  means  for  the  instructor  to  know  if  the  student  is 
ready  to  be  tested? 

_8.  Have  you  determined  how  long  a  student  can  spend  on  a  particular  test? 

_9.  Have  you  determined  if  there  is  to  be  a  minimum  time  to  spend  studying 
between  test  and  retest? 

10.  Have  you  determined  how  long  a  student  may  spend  on  a  particular  lesson? 

1 1.  Have  you  set  the  criterion  for  passing  a  lesson? 

12.  Have  you  devised  a  means  for  the  instructor  to  know  where  the  student  is 

located  in  the  program? 

13.  Have  you  determined  what  the  instructor  will  do  if  a  student  is  too  slow? 

14.  Have  you  determined  what  the  instructor  will  do  if  a  student  has 
"free  time?" 

15.  Have  you  provided  a  means  for  the  instructor  to  know  if  a  student  needs 
extra  help? 

16.  Have  you  provided  a  means  for  the  student  to  know  if  he  or  she  needs 
extra  help? 

17.  Have  you  provided  a  means  for  a  student  to  get  extra  help? 

18.  Have  you  determined  how  long  a  student  can  take  to  complete  the  course? 

19.  Have  you  set  the  criterion  for  passing  the  course? 
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EXPERT  TRYOUT  QUESTIONNAIRE 


1.  Do  you  think  you  answered  each  item  correctly? 

A.  Yes.  B.  No.  C.  Not  sure. 

(Go  to  4.)  (Go  to  2.)  (Go  to  3.) 

2.  Which  items  do  you  think  you  did  not  answer  correctly? 

3.  Which  items  are  you  unsure  about? 

4.  Is  this  test  realistic?  (Does  it  reflect  how  the  job  is  done?) 

A.  Yes.  B.  No.  C.  Not  sure. 

5.  Were  the  words  or  phrases  in  the  test  easy  to  understand?  (If  "No,"  indicate  in  the 
space  below  the  words  or  phrases  that  gave  you  trouble.) 

A.  Yes.  B.  No.  C.  Not  sure. 

6.  Are  all  of  the  alternatives  to  each  item  believable?  (If  "No,  indicate  the  alternatives 
that  are  not  believable.) 

A.  Yes.  B.  No.  C.  Not  sure. 

7.  Are  all  the  items  relevant  to  the  job?  (If  "No"  or  "Not  sure,"  indicate  the  items  that 
should  be  deleted.) 

A.  Yes.  B.  No.  C.  Not  sure. 

8.  As  the  test  is  now  written,  do  you  think  it  is  a  fair  measure  of  a  student's  ability  to 
perform  on  the  job? 

A.  Yes.  B.  No.  C.  Not  sure. 
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SUGGESTED  ACTIONS  IF  TEST  IS  NOT  ACCEPTABLE  TO  EXPERTS 


Question 

Answer 

Possible  Actions 

1 

B  or  C 

a. 

Check  that  answer  is  correct. 

b. 

Check  "wrong"  alternatives  are  incorrect, 

c. 

Rewrite  stem. 

d. 

Rewrite  situation. 

e. 

Rewrite  alternatives. 

2 

N/A 

3 

N/A 

ft 

B 

a. 

Check  that  answer  is  correct. 

b. 

Add  illustration. 

c. 

Rewrite  situation. 

d. 

Consider  hands-on  testing. 

5 

B 

a. 

Rewrite  entire  item. 

b. 

Add  illustration. 

6 

B 

a. 

Change  unbelievable  alternatives. 

7 

B 

a. 

Delete  entire  item. 

b. 

Rewrite  situation. 

8 

B  or  C 

a. 

Probe  for  more  information  on  preceding 

questions. 


TABLE  FOR  RECORDING  SCORES  TO  VALIDATE 
PAPER-AND-PENCIL  TESTS 


HANDS-ON  TEST  QUESTIONNAIRE 

Yes 

1.  Is  the  task  covered  by  the  test  a  job  requirement?  _ 

If  not,  why  not? 


2.  Does  the  test  provide  a  fair  measure  of  job 
requirements? 

If  not,  what  changes  are  required? 


a.  Add  performance  measures? 
Which  ones? 


b.  Delete  performance  measures? 
Which  ones? 


c.  Modify  performance  measures? 
How? 


d.  Alter  standards? 
How? 


e.  Clarify  instructions? 
How? 


f.  Change  test  site  or  equipment  requirements? 
How? 
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DISTRIBUTION  LIST 


Chief  of  Naval  Operations  (OP-987H) 

Chief  of  Naval  Education  and  Training  (02),  (N-2),  (N-5) 

Chief  of  Naval  Teechnical  Training  (016) 

Commandant  of  the  Marine  Corps  (MPI-20) 

Commandant  of  the  Marine  Corps  (TDE) 

Commander  Naval  Ocean  Systems  Center  (Code  033) 

Commander  Training  Command,  U.S.  Atlantic  Fleet 
Commander  Training  Command,  U.S.  Pacific  Fleet 

Commanding  General,  Marine  Corps  Development  and  Education  Command 
Commanding  General,  Marine  Corps  Development  and  Education  Command,  Instructional 
Management  School 

Commanding  General,  Landing  Force  Training  Command,  Pacific,  Instructional  Manage¬ 
ment  School 

Commanding  General,  Instructional  Management  School,  Marine  Corps  Service  Support 
School 

Commanding  Officer,  Naval  Education  and  Training  Program  Development  Center  (Tech¬ 
nical  Library) (2) 

Commanding  Officer,  Naval  Education  and  Training  Support  Center,  Pacific 
Director,  Defense  Activity  for  Non-Traditional  Education  Support 
Director,  Management  Information  and  Instructional  Activity  Branch  Office,  Memphis 
Director,  Naval  Education  and  Training  Program  Development  Center  Detachment,  Great 
Lakes 

Director,  Naval  Education  and  Training  Program  Development  Center  Detachment, 
Memphis 

Director,  Training  Analysis  and  Evaluation  Group  (TAEG) 

Officer  in  Charge,  Central  Test  Site  for  Personnel  and  Training  Evaluation  Program 
Commander,  Air  Force  Human  Resources  Laboratory,  Williams  Air  Force  Base  (CNET 
Liaison  Office  AFHRL/OTLN) 

Commandant  Coast  Guard  Headquarters 
Superintendent,  U.S.  Coast  Guard  Academy 


43.  An  Important  general  rule  to  follow  when  constructing  process-scored  measures  is  to: 

a.  Put  only  a  single  action  into  each  measure. 

b.  Put  all  safety  performance  measures  at  the  beginning  of  the  test. 

c.  Avoid  the  use  of  scoring  aids. 

d.  Include  as  many  actions  as  possibie  in  the  test. 
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Pass 


9 


5 


Fail  6  12 

Formulas  (A  x  D)  -  <B  x  C)  =  Agreement  index 
Agreement  index  =  _ _ — - — 


B-24 


Diagnosis 

Prescription 

Date/Tlme 

N 

INSTRUCTION 

- 

V 

Date/Tine 

Ipstr. 

Instr. 
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